Digital picture frames and methods of photo sharing

ABSTRACT

A picture frame and methods of use. The picture frame includes a housing with a display area on a first side and a back panel on an opposite second side from the display. The back panel includes an asymmetrical nonplanar shape and/or an off-center rotatable stand. A network connection allows digital frames to upload of photos from electronic devices or from photo collections of community members. An integrated camera is used to automatically determine an identity of a frame viewer and can capture gesture-based feedback. The displayed photos are automatically shown and/or changed according to the detected viewers. The photos can be filtered and cropped at the receiver side. Clustering photos by content is used to improve display and to respond to photo viewer desires.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 16/113,366, which is a continuation-in-part of U.S. patentapplication Ser. No. 15/824,337, filed on 28 Nov. 2017, which is acontinuation-in-part of U.S. patent application Ser. No. 15/150,623,filed on 10 May 2016, which is a continuation-in-part of U.S. patentapplication Ser. No. 14/949,353, filed on 23 Nov. 2015, which is acontinuation-in-part of U.S. patent application Ser. No. 14/455,297,filed on 8 Aug. 2014, which is a continuation-in-part of each of: U.S.patent application Ser. No. 14/051,071, filed on 10 Oct. 2013, and U.S.patent application Ser. No. 14/051,089, filed on 10 Oct. 2013. Theco-pending parent application(s) are hereby incorporated by referenceherein in their entirety and are made a part hereof, including but notlimited to those portions which specifically appear hereinafter.

FIELD OF THE INVENTION

This invention relates generally to a digital picture frame and, moreparticularly, to a digital picture frame that is a Mobile PositionalSocial Media (MPSM) output device that automatically providespersonalized photo viewing in automated combination with a proximate orremote electronic device and/or an MPSM account,

BACKGROUND OF THE INVENTION

Existing digital picture frames synchronize with popular social mediaplatforms, such as Facebook™ and Instagram™, using, among othertechniques, a WiFi™ interconnection. However, multiple steps may berequired for such synchronization. These manual steps can be difficultto do if one wants to change to a different social media account or todirectly retrieve from a picture capture or storage device, such aswhile entertaining guests. There is a continuing need for improvementsfor implementing displays such as home digital picture frames.

SUMMARY OF THE INVENTION

The present invention provides a picture frame for displaying pictures.The picture frame is desirably a digital picture frame thatautomatically detects viewers and/or viewers' electronic devices, andautomatically displays photos or videos relevant to the detectedviewers. The relevant photos or videos are obtained upon detection fromthe electronic device and/or social media accounts of the detectedviewers. The photos or videos can also be automatically obtained fromdevices, frames, stored photo collections, or social media accounts ofone or more community members of the detected viewer(s), such as photostaken and posted by a friend or relative of the detected viewer. Thedevice and method of this invention provides for improved and updatedphoto display based upon current photos and/or social media updates, andthus relevant to the viewer. The device of this invention can be used asan MPSM device to display any MPSM information, as desired, includingbut not limited to MPSM information obtained according to methodsdescribed herein.

The invention includes a picture frame including a display and a housingincluding the display on a first side and a back panel on an oppositesecond side from the display. The back panel includes an asymmetricalnonplanar shape. The display is desirably a digital display, but caninclude a printed picture display area as well.

In embodiments of this invention, the back panel has an obliquepyramidal shape. The oblique pyramidal shape can extend outward from orbe recessed into the housing. The back panel can include four irregulartrapezoidal sections, each having a different size and shape.Embodiments of this invention include a power port within the obliquepyramidal shape. The four trapezoidal sections can disposed about apower port.

Embodiments of this invention include an off-center rotating stand, suchas centered between the four irregular trapezoidal sections. Therotating stand can be disposed over a power port within the back panel.In embodiments, the stand rotates to sit the frame upon either of a pairof perpendicularly abutting frame sides. The frame can include a touchbar disposed on each of a second pair of perpendicularly abutting framesides. The touch bar can be manually used to controls a digital photochange on the display.

The invention further includes a picture frame with a back panel thatincludes an off-center rotatable stand. The back panel can includes anasymmetrical nonplanar shape, such as described above. An off-centerpower port is desirably recessed within the stand attachment, whereinthe stand is disposed over the power port. A stand attachment desirablyincludes a two direction slide mechanism or a rotating joint by whichthe stand rotates to sit the stand upon either of a pair ofperpendicularly abutting frame sides. The frame can include a touch bardisposed on each of a second pair of perpendicularly abutting framesides. A stop mechanism can be configured to limit a rotation of thestand to 90 degrees. In embodiments of this invention, a rotationlimiter is within the stand attachment on the back panel.

Embodiments of the invention include a digital picture frame with adigital display mounted within a frame, a camera connected to the frame,a network connection module adapted to connect to an electronic device,photo collection, and/or social media account of a viewer viewing thedigital picture frame and to receive photos stored within the electronicdevice and/or the social media account, and an automated display moduleadapted to automatically change photos displayed on the digital displayto photos from the electronic device and/or social media account of theviewer, upon automatic detection with the camera of the viewer.

The invention includes a method of displaying photos on electronicdevices with a network connection, including: automatically obtaining adigital photo from a first electronic device; automatically identifyinga content of the digital photo; automatically identifying a targetdisplay electronic device as a function of the content; andautomatically routing the digital photo over the network connection tothe target display electronic device for display. The first electronicdevice is controlled by a first community member and the target displaydevice, such as a mobile device or a digital picture frame, can becontrolled by a second community member. The target display electronicdevice desirably includes predetermined content constraints, and thedigital photo is routed and/or displayed if the photo meets the contentconstraints.

The invention further includes a method of displaying photos onelectronic devices with a network connection via: automaticallyobtaining digital photos on a first electronic device; automaticallyidentifying a content of each of the digital photos; automaticallymatching the content of a first of the digital photos to a predeterminedcontent constraint of a second electronic device; and automaticallyrouting the first of the digital photos to the second electronic deviceover the network connection for display on the second electronic device.The method can further include automatically processing the digitalphotos by at least one of image enhancement, content filtering, and/orrepetition filtering. The content filtering can include identifying theplace, person, and/or activity, etc. of the photo, which can then becompared to preferred content (e.g., a stored metadata identificationset) of the potential target/receiving frames.

In presently preferred embodiments of this invention, where the contentof the first of the digital photos matches a predetermined contentconstraint of a plurality of community members, the first of the digitalphotos is automatically routed to electronic devices of each of thematching community members. The invention further includes a method ofdisplaying photos on electronic devices with a network connection,including: automatically obtaining digital photos on a first electronicdevice; automatically identifying a content of each of the digitalphotos; automatically matching the content of each of a plurality of thedigital photos to a predetermined content constraint of a correspondingelectronic device of one or more community members; and automaticallyrouting each of the plurality of digital photos to the correspondingelectronic device as a function of the matching for display on thecorresponding electronic device.

The invention further comprehends a method of displaying photos on adigital picture frame including a digital display mounted within aframe, a camera connected to the frame, and a network connection module.The method includes: automatically determining with the camera anidentity of a viewer of the digital picture frame; automaticallydetermining a viewer profile from the identity of the viewer; andautomatically changing photos shown on the digital display as a functionof the viewer profile upon detection of the viewer with the camera. Themethod can include automatically augmenting a photo display on thedigital display as a function of photo metadata and the viewer profile.In embodiments of this invention, the viewer profile is automaticallylearned and uploaded via an electronic device and/or a social mediaaccount of the viewer.

The invention further comprehends a method of displaying photos on adigital picture frame including a digital display mounted within aframe, a camera connected to the frame, and a network connection module.The method includes automatically determining with the camera anidentity of a viewer of the digital picture frame, automaticallyconnecting to a picture capture or storage electronic device of theviewer over a network, such as a WiFi network, and automaticallydisplaying photos shown on the digital display obtained from theelectronic device over the network connection.

The device and method of this invention can automatically display aslideshow of photos relevant to one or more display viewers as afunction of photo context selected from time taken, photo location,and/or photo content. The device can automatically access, using thenetwork module, at least one electronic device, digital photocollection, and/or social media account of: the viewer and/or at leastone social media community member of the viewer. Photos from the one ormore electronic devices, photo collections, or social media accounts aredesirably automatically loaded and displayed. In embodiments of thisinvention, obtained photos are automatically sequenced on the digitaldisplay as a function of profiling traits selected from chronologicalorder, photo location, photo activity, and/or community member.

The device and method can detect and recognize two or more different andsimultaneous, or otherwise present, persons viewing the digital pictureframe, and automatically change the photos displayed on the digitaldisplay to photos uploaded from an electronic device and/or a socialmedia account of each of two or more persons viewing the digital pictureframe. The photos from the electronic device or social media account ofeach of the two or more persons can include shared activity photos fromactivities shared by the two or more persons. The invention includesembodiments where a server computer automatically determines sharedactivity photos from the electronic device and/or social media accountof the each of the two or more persons as a function of contextinformation automatically associated with the shared activity photos bythe same or different server computer. The context information of eachof the shared activity photos can include, but is not limited to, aphoto location, a photo activity, and/or a present community member.

The invention still further comprehends a display device with one ormore of: automatic detection for configuration, namely theself-straightening or self-positioning on a vertical (e.g., wall) and/ora horizontal (e.g., table) platform; the automatic detection of viewingconditions, namely, based on the distance of the viewer, enlarging orcontracting the picture; based on the lighting, increasing or decreasingthe lighting intensity. The invention also comprehends a display devicethat: via face recognition, automatically detects the viewer and targetsthe photos for her or him based on their potentially learned profile;via the automatic identification and tagging of the location, activity,and community member involvement on a per photo or video, indicating thewith whom, where, when and what was being done when the picture wastaken as well as any other associated metadata; and/or providing a“story telling” capability. By storytelling, chronological stories,optionally simultaneously displayed on a split screen, are grouped by:purely time, namely in sequential or reverse chronological ordering;location, namely a traversal of sites on a location based trip;activity, namely in chronological ordering of a given or similar set ofactivities; community member involvement, namely a pictorial interactionwith community members, potentially segmented by particular communitymember or members; or any other profiling traits of a recognized userthat can be used to cluster or segment photos or videos for automaticstory telling.

The invention includes a method of displaying photos on a digitalpicture frame and including: automatically determining with the cameraan identity of a viewer of the digital picture frame; automaticallyconnecting to one or more digital photo collections over a network; andautomatically displaying on the digital display photos obtained from thedigital photo collection over the network connection. The digital photocollection(s) can be associated with a second digital picture frameand/or an electronic device of a community member of the viewer. Thedigital picture frame has access to the digital photo collection(s) anda server computer automatically sends the photos to the frame throughthe network connection module. The server or frame desirablyautomatically crops and aligns the photos to fit the digital display,such as by automatically scaling and realigning the photos to an aspectratio of the digital display.

Additional automated processing can include image enhancement, contentfiltering, or repetition filtering. The photos can be filtered by imagequality and content quality. The digital picture frame can include acontent restriction filter module including adjustable contentrestraints controlled by a frame administrator, for implementing contentrestrictions of the viewer obtained from a viewer profile over thenetwork. The frame administrator and/or the viewer can have or setpre-established filtering parameters selected from image quality and/orcontent quality, for use in filtering the digital photo collection.

In embodiments of this invention, the device and method includeautomatically detecting via the camera a physical response to one of thedisplayed photos, and automatically assigning an interest level grade tothe one of the displayed photos using the physical response. Thephysical response can be a body movement or a facial expression. Theinterest level grade can be stored in a viewer profile for the viewer,and used for displaying photos for the viewer as a function of theinterest level grade. The physical response, positive or negative, canbe used to identify and suggest other related photos to the viewer, suchas for a replacement photo related to the same subject matter.

In embodiments of this invention, the device and method includeautomated ranking of the photos to be circulated on the frame, such asfor automatically determining an order of presentation, a number ofdisplay repetitions, and/or a time period of display for each of thephotos as function of the ranking. The ranking can be based at least inpart on a physical feedback response of the viewer to one or more of thephotos, or based upon a picture quality and/or content. The photos canbe clustered according to the rankings into groups for display orsharing.

The invention further includes a method and device for displaying photoson a digital picture frame that includes: automatically clusteringphotos of one or more digital photo collections for display to theviewer, and automatically displaying on the digital display the photosfrom the digital photo collection as a function of the clustering. Theclusters are groups of photos to form photo collections of relatedcontent, such as selected from people, things, places, activities, orcombinations thereof, and displaying the photo collections. The photosare desirably clustered by a photo content, which can be identified orconfirmed by photo metadata. Clustering of photos with a preselectedphotographed person can be performed by automatically identifying andclustering the photographed person by facial features. Clusteringaccording to this invention is particularly useful for photos taken overa multi-year timeframe, and grouping photos by related photo contentacross the timeframe. Clustered photos can be used to send photos to acommunity member frame, such as by clustering in an app on a mobiledevice, and selecting, within the app, a cluster and a community memberframe to send the photos of the photos to the community member frame.For example, a son's photos can be clustered by faces, and clusters ofthe son's children (the grandchildren) can be sent as a cluster to hismother's frame (the grandmother) from the son's mobile device app.

The digital picture frame of embodiments of this invention operates inconcert with a method, system, and apparatus, such as embodied in anMPSM or other software application, that automatically determines andshares a location and/or an activity of a user. The application learnsuser activity over time, with the learning based upon user locationsand/or context. The present invention generally provides methods andapplications for an MPSM that automatically understands and informs the“who, what, when, where, and/or how” of a user and the user's community.For example, who are the user and their community with, what are theuser and their community doing, where geographically are the user andtheir community, when are, and historically when were, the user andtheir community doing this, and/or how can users' behaviors be modified?The method includes automatic tagging of photos taken by the users withcorresponding context data (e.g., location, activity, people present)learned by the system for use in determining photos to display on thedigital picture frame of this invention as discussed above.

The invention includes a method of learning and sharing locations and/oractivities, and pictures generated therefrom, of a user participating ina social networking service. The method is executed by a softwareapplication stored and executed on one or more computers or dataprocessor systems, such as a mobile device (e.g., phone, tablet, orlaptop) and/or an application server (such as for connecting usercommunities). In one embodiment the method includes receiving userinformation about a destination, automatically associating the userinformation with the destination, and automatically sharing the userinformation in the social networking service upon further user arrivalsat the destination prior to receiving any additional user information.

The invention still further includes a computer server for providing atracking and/or social networking service, such as operating the methodsdiscussed herein. The computer server of one embodiment of thisinvention includes a tagging module configured to correlate userinformation and/or pictures thereof to a user destination or activity, adatabase module configured to store user information including userlocations and user activities at the user locations, and a communicationmodule configured to automatically share a user activity in the socialnetworking service upon further user arrivals at a corresponding one ofthe user locations. The computer system can also include an associationmodule configured to associate the user activity with the correspondingone of the user locations.

In embodiments of this invention, the system or application identifieslocations, and over time, automatically “checks-in” not only thelocations but what the locations imply in terms of potential activitiesof the user. That is, given a location and a user, the system desirablysuggests what activity or activities the user typically partakes at thatlocation. For example, if a user frequents a location in “Potomac,” thislocation might be identified as “parents' home.” Furthermore, at thishome, a variety of activities might be common such as: “visitingparents,” “drinking tea,” “eating lunch,” or “sampling wine.” Inembodiments of this invention, each time the user appears at thatlocation, based on context, defined by elements of or surrounding theactivity such as but not limited to time of day, day of week,immediately preceding activities, weather, surrounding people, etc., aset of likely occurring activities are identified. The user can beprompted with a list from which to select a subset of these activitiesor to identify a new activity. The invention can also include ranking auser's potential suggested activity based on context and presenting thatranked list to the user, or the user's community. The inventiongenerally provides a learning component that can allow the manual inputsto become automatic prompts, which can become automatically issuednotifications for the location based upon the context. The prompts canbe issued through any known format, such as an application alert on thedevice or a text message to the user. The invention also supports theuser changing activities for a given location at any time, and/or userimplemented delay of the notification of a user's location or activity.

The invention can include the incorporation or creation of usercommunities and sub-communities, with such communities andsub-communities sharing information. Embodiments of the inventioninclude automatically identifying a user's location and activity, anddesirably notifying that user's community of that user's location, aswell as sharing relevant photos from the user's community to the digitalpicture frame. Particular embodiments of this invention provide one ormore additional community functionalities including, without limitation,automatically identifying a user's activity and notifying that user'scommunity of that user's activity, commenting on user activity andlocation report by user or community—with multiple and multimediacomments supported, supporting the “liking” of user activity by thecommunity, supporting the user tagging of location, activity, or thepairing of location and identity—tagging can be textual or via anymultimedia means, correlating the individual user's activity with theongoing activity of others within the community, and/or correlating theindividual user's activity with the past activities of others within thecommunity, all useful in generating user profiles and tagging photos forautomated display on the digital picture frame of this invention.

One embodiment of this invention provides a method of and system forautomated determining of locations and/or activities of a userparticipating in a social networking service. The method is executed byan MPSM computer system and automatically determines a positionaldestination of a user, automatically deduces as user information alocation type and/or user activity of the positional destination, andautomatically shares the user information as instructed in the socialnetworking service. Deducing the user information is based upon contextinformation about the positional destination, desirably with minimal orno input by the user. The context can include, without limitation,time-dependent information, past and/or current associated userinformation, past user and/or community information about the location,and/or third party information. The context can be used to at leastreduce location types and/or user activities, for example, as a functionof the past location type and/or user activity of the positionaldestination for a given time period.

The invention further includes marketing methods for the frame thatmotivate multiple purchases. First, volume discounts are provided forthe simultaneous purchasing of multiple frames. Additionally, the timeof the most recent purchase is noted, and should another purchase by thesame customer occur within a predetermined, but possibly varying, amountof time, a discount is applied. The duration might be based on theseason, sales history, stock availability, customer identity, or anyother business driven rule.

Other objects and advantages will be apparent to those skilled in theart from the following detailed description taken in conjunction withthe appended claims and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a digital picture frame in network connection to aserver and mobile devices, according to one embodiment of thisinvention.

FIG. 2 illustrates a digital picture frame in network connection to aserver and mobile devices, according to one embodiment of thisinvention.

FIG. 3 shows a back view of a picture frame according to one embodimentof this invention.

FIGS. 4A-4D illustrate a picture frame and use according to oneembodiment of this invention.

FIG. 5 shows a rear perspective view of a picture frame according to oneembodiment of this invention.

FIG. 6 shows a stand attachment according to one embodiment of thisinvention.

FIGS. 7A-7B shows a rear perspective view of a picture frame accordingto one embodiment of this invention.

FIGS. 8 and 9 illustrate a photo clustering according to one embodimentof this invention.

FIG. 10 shows a representative area of a user according to an MPSMmethod of one embodiment of this invention.

FIG. 11 illustrates geofences surrounding a current reading and itsimmediate neighbors according to one embodiment of this invention.

FIG. 12 illustrates the determination of a location via intersectingcircles according to one embodiment of this invention.

FIG. 13 illustrates the processing flow employed to identify an arrivalaccording to one embodiment of this invention.

FIG. 14 illustrates a system view location summary of an individual useraccording to one embodiment of this invention.

FIG. 15 illustrates a system view transit summary of an individual useraccording to one embodiment of this invention.

FIG. 16 illustrates a system view activity summary of an individual useraccording to one embodiment of this invention.

FIG. 17 illustrates a system view time location breakdown in comparisonto other users according to one embodiment of this invention.

FIG. 18 illustrates a system view a listing of activities shared withother users according to one embodiment of this invention.

FIG. 19 illustrates a view presented to users that quantifies theirshared patterns according to one embodiment of this invention.

FIG. 20 illustrates a view presented to users summarizing their weeklybehavior according to one embodiment of this invention.

FIG. 21 illustrates a view presented to a user of her/his and her/hiscommunity's activities in a pictogram format, according to oneembodiment of this invention.

FIGS. 22-24 illustrate various pictogram summaries for user activities.

FIGS. 25-27 illustrate adding photos to pictograms and changingpictograms.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides a digital picture frame that interactswith either a picture capture or storage device, referred to herein asan electronic device, running a digital picture frame application and/orsocial media to automatically display digital photographs from theelectronic device and/or a social media account of one or more personsviewing the picture frame. As used herein, the terms “picture”,“photograph”, or “photo” refer to or include digital still photographsor digital video, including a combination of few-second animationseither or both before and after a still frame, unless otherwise noted.The photos can be obtained by social media users and stored in theusers' social media accounts. When a social media user views the digitalpicture frame, the digital picture frame automatically recognizes thatuser and automatically loads photos relevant to that viewer from her orhis electronic device and/or social media account and/or from at leastone electronic device and/or social media account of her or his socialmedia community member(s). For example, relevant photos from a communitymember's social media account may include the viewer or be from a sharedactivity that the viewer attended with the community member. Automatedsocial media activity learning methods, as described herein, are used toautomatically determine relevancy of photos to one or more communitymembers. Community members may choose to make photos shared or privateto avoid personal photos being shared.

FIG. 1 illustrates a digital picture frame 20 according to oneembodiment of this invention. The digital picture frame 20 includes adigital display 22 mounted within an edge frame 24. The digital display22 can incorporate any suitable screen, such as, without limitation, anLED screen or a touch screen, as are known and commercially available. Acamera 25 is connected to and integrated in the edge frame 24. Thepicture frame 20 includes a network connection module that connects overa wired and/or wireless network to a social media server computer 26 andcan automatically load photos stored within one or more social mediaaccounts accessed through the server 26. The picture frame 20 furtherincludes an automated display module in combination with the networkconnection module and adapted to automatically change photos displayedon the digital display 22 to photos from an electronic device and/or asocial media account of the viewer and/or one or more community members,such as upon automatic detection with the camera of the viewer. Thedigital picture frame 20 desirably includes a microphone 27 incombination with the automated display module and/or network connectionmodule, such as for receiving spoken instructions from a viewer tochange the displayed photos.

The digital picture frame 20 further includes a leveling device 28connected to or otherwise integrated with the frame 24. The levelingdevice 28 automatically detects when the frame is not level. Properpositioning of the digital picture frame is important to achieving usersatisfaction. Auto detection of the positioning in comparison to theangle of the displayed photograph illustrated can be achieved using anyof the image alignment or realignment techniques known in the art. Inembodiments of this invention, the leveling device 28 can transmit ordisplay corrective measures for physical correction.

In another embodiment of the invention, the digital picture frame 20 isconfigured with a leveling device that includes one or more motors 23,such as paired attached with a corresponding wall bracket via gearing.The motors 23 can be incorporated in a wall hanging mechanism as shownin FIG. 1, or otherwise implemented with respect to a frame stand. Upondetection of an unleveled viewing as determined by either the levelingdevice 28 or any other mechanism, the dual motors 23 activate andautomatically mechanically operate as needed to level the digitalpicture frame 20, such as a vertical adjustment on wall mountinghardware, via the adjustable hanging mechanism or bracket.

The digital picture frame 20 includes an automated display adjustmentdevice in detection combination with the camera 25. The automateddisplay adjustment device automatically detects frame environment and/orviewer position using the camera 25 and automatically adjusts settingsof the digital display 22. Augmented with the camera 25, the digitalpicture frame 20 supports a host of optional, additional capabilities.Using conventional, known in the art, image processing techniques,enhancement of the picture quality can be automatically made. Forexample, due to an automatically detected viewing angle or environmentalbrightness, picture lighting intensity and contrast can be altered. Thatis, depending on conditions, the brightness and contrast can beautomatically modified to provide better viewing. Additionally,depending on the distance and angle of the viewer, the size of the imagecan be enlarged, possibly focusing on the center or content rich area ofthe picture. Similarly, a close viewer may wish to have the photographor video in full scope, or maybe, even side-by-side with one or moreother, possibly related pictures or videos.

In one embodiment, as shown in FIG. 1, the digital picture frame 20includes the capability to ‘daisy chain’ one or more additional frames20′. That is, all participating frames can be connected so as to portraya single story, either chronologically or thematically, allow for twoviewers, and/or provide a composite view of a single image. The frame 20can include a network connection 10, such as an audio-video outletand/or network outlet, to connect to a corresponding inlet 12 of thesecond frame 20′. The network connection between frames can be by wiredconnection (e.g., HDMI cord or Ethernet cable) or wireless communication(e.g., Bluetooth or WiFi) The frame 20 can optionally include a poweroutput, such as outlet 14 to receive a power cord 16 of the second frame20′, so as to require only a single power source.

An embodiment of a digital picture frame 20 further includes multiplepower options. In addition to conventional outlet and battery options,the frame can include at least one photoelectric cell. Inductive, orwireless, charging can also be incorporated.

The digital picture frame 20 includes a facial recognition module withany suitable facial recognition software, in combination with the camera25 and automated display module. Face recognition software can bedeployed to automatically detect the identity of the viewer(s). It isalso within the scope of this invention to identify the viewer via thelocation or proximity of their personal electronic devices such as butnot limited to their mobile phones or tablets. By identifying theviewer(s) the automated display module can then specially target, namelyselect, prioritize, and/or order the photos or videos displayed for theidentified viewer.

Identifying the viewer enables a variety of novel applications. Inembodiments of this invention, a viewer analysis module is combined witha facial recognition module to determine, for example, one of mood orhealth of the viewer upon detection of the viewer with the facialrecognition module. The automated display module can augment a photodisplay on the digital display as a function of the determined mood orheath of the viewer, such as by showing photos to increase the viewer'smood.

Longitudinal analysis of the viewer is possible. Such analysis supports,for example, health commentary. By analyzing the viewer's face, commentssuch as, but not limited to, “you seem to have gained weight”, “you looktired”, or “you seem pale; do you feel well?” can be announced to theviewer. Similarly, “mirror, mirror on the wall” remarks such as “youlook good today” can be stated, potentially elevating the viewer's mood.Displaying identified mood enhancing photographs, such as, for example,ones that include far away loved ones can likewise be shown to furtherelevate the viewer's mood. Identification of mood enhancing photographscan be automatically accomplished, for example, via the recognition of a“smiling face” or spoken words of the viewer when the picture waspreviously shown, and tagging of the photo.

Viewer identification also enables photo selection personalization. Aviewer personalization module can be operatively combined with thefacial recognition module, whereby the viewer personalization moduleautomatically identifies preferred photos or restricted photos as afunction of viewer information upon detection of the viewer with thefacial recognition module. Photos favored by the viewer (e.g.,grandchildren for the grandparent) can be selected, and converselyphotos containing, for example, ex-spouses or in-laws can be repressed.Likewise, identifying viewer characteristics, such as age, enables thelimiting of photos displayed. For example, children might not be shownphotos containing potentially inappropriate content, such as containingnudity, or particular activities or individuals, such as relatives thathave recently passed away. Thus, it is within the scope of thisinvention to limit or specifically select photographs to be displayedbased on relationships of the viewer with individuals in thephotographs, such as ex-spouses; characteristics of individuals shown inthe photographs, such as babies; characteristics of the viewer, such asage; activities illustrated in the photographs, such as smoking;locations displayed in photographs both physically described (e.g.,Chicago) or relatively described (e.g., viewer's place of residence); orbased on any characteristics that can be determined via image analysisor photograph metadata included. The filtration or selection ofphotographs based on criteria is determined by settings that can be set.

Additional specialty settings are included. Such setting place the framein nightlight mode where only a dim glow is illustrated; photo recordmode to simply record all in the line of vision; video chat mode withother devices; and “stranger in the house” mode where only limited orstock photos are shown to prevent the potential identification ofresidents.

The digital picture frame 20 can likewise be used as a viewer assistant.Multiple ailments or conditions, such as but not limited to Alzheimer's,strokes, and dementia, affect memory. Thus, the digital picture frame 20can be used to remind the viewer of the identity of family members andfriends as well as of occasions, locations, activities, etc., ofrelevance as the photos displayed are labeled both by content and/or bymetadata.

As previously discussed, the network connection module connects over anetwork connection to the social media server computer 26. It canlikewise connect either directly or indirectly to the mobile device 29′of any community member or interested party having an appropriatelyconfigured device, such as digital picture frame application specificsoftware. The connection need not be within a geographic proximity, andit is likewise within the scope of this invention that a geographicallyremote connection is supported via networks such as but not limited toas the Internet.

The social media server computer 26 can be any one or more computersthat implement any suitable social media platform, such as the MPSMplatform and automated learning steps described herein. The servercomputer 26 can obtain or have access to photos and learn user oractivity information associated with each photo from, for example,mobile devices 29 of a plurality of MPSM users or community members,such as according to methods described herein. In embodiments of thisinvention, the automated display module augments a photo display on thedigital display 22 as a function of photo metadata from the social mediaaccount stored on the server computer 26. The photos are automaticallydownloaded from the server 26 to the digital frame 20 upon, for example,automated detection of a corresponding viewer(s).

In embodiments of this invention, the photos can be changed according tothe learned bias of a viewer, expressed and stored as part of a userprofile at the server computer 26 and/or the digital picture frame 20.The picture frame 20, such as via the automated display module, includesor obtains the viewer profile of the viewer upon detection of the viewervia the camera. By establishing a viewing profile for each known orexpected viewer, e.g., if a viewer is or has a community member,individual or group user viewing preferences can be maintained. Aprofile can be established using any one or more of the many knownprofile techniques known in the art, including but not limited to,asking the viewer, learning based on previous user selections, and/orcorrelations with previously automatically learned, via MPSM methodsherein, locations, activities, and/or community member interactions.

In a preferred embodiment, viewer profiles are automatically learnedaccording to methods described herein, and uploaded via an electronicdevice and/or the social media account of a detected viewer and/or heror his community members. The automated display module desirablydisplays a slideshow of photos for a detected viewer that is uploadedfrom: an electronic device of the viewer, and/or a social media accountof the viewer or one or more social media community member of theviewer.

In the digital picture frame embodiments of this invention pictures orvideos displayed can be augmented with the automatically learned andidentified metadata corresponding to the pictures or videos. That is,the information learned, captured and/or displayed identifies thelocation, activity, and community member involvement on a per photobasis, indicating the with whom, where, when and what was being donewhen the picture or video was taken as well as any other associatedmetadata, such as the camera or video equipment used to capture thephoto or video. Slideshow of photos for a viewer can be assembled andshown in a display order automatically determined as a function of photoprofiling traits such as time (e.g., chronological order), photolocation, photo content (e.g., people or activity shown in the photo),and/or community member presence at the time/location that a photo wastaken.

The invention includes a method of displaying photos on a digitalpicture frame, such as shown in FIG. 1. The digital picture frame ishung or set on a surface by a user, for explanation purpose, namedSusan. The picture frame automatically determines with the camera whenSusan is viewing, or is the primary viewer of, the picture frame. Thepicture frame automatically determines Susan's profile from herdetection. In embodiments of this invention, Susan's viewer profile isloaded from Susan's electronic device and/or social media account, andmay be supplemented by additional information gathered via the pictureframe itself, such as face recognition, account access information,and/or manual entry (e.g., picture frame display preferences). Theviewer profile is desirably created and continually updatedautomatically according to the automated learning methods describedherein. The viewer profile can include information such as, withoutlimitation, family information, community member information, and/oruser favorites (e.g., favorite locations, activities, colors, flowers,animals, etc.), learned through the social media account.

When the camera and coordinated software determines Susan is viewing thepicture frame, e.g., either directly or merely in the vicinity, thepicture frame automatically displays photos shown on the digital displayas a function of the viewer profile information. The photos displayedare automatically augmented or picked as a function of comparing photometadata and the viewer profile. In embodiments of this invention, thepicture frame automatically displays for Susan a slideshow of photos(still and/or videos) relevant to Susan as a function of photo context.The photo context can be selected from the time the photo was taken(e.g., all photos from July 2015), photo location (e.g., all photostaken in Chicago), and/or photo content (e.g., all photos of vacationsor involving her sister and/or her friend Mary). The photos can beimported primarily from her electronic device and/or social media, andoptionally supplemented from photos on a recordable medium of the frame(e.g., hard drive or inserted flash drive or memory card) or photosobtained from a third party web site.

It is within the scope of this invention to prompt the user, possiblyvia any communication mechanism including but not limited to text andvoice commands, to determine the exact wishes of the user. If thedigital picture frame is additionally equipped with a microphone 27 andcorresponding speech processing software, voice commands can likewise beprocessed using any of the many known speech processing systems. Susancan select a context, such as by spoken instructions, to be display. Asan example, Susan may request all photos related to “college”, relatedto “vacations”, including “family”, or including or relating to aparticular person. The digital picture frame will download relevantphotos for display from one or more electronic devices and/or socialmedia accounts, of Susan's and/or a community member of Susan, that havecontext matching the request. If Susan asks for pictures of her and hersister, the pictures may include her sister or have been taken in thepresence of both her and her sister, such as a picture of the EiffelTower taken during a trip with her sister to France. The Eiffel Towerphoto is identified by context metadata that identifies that both Susanand her sister were present when the photo was taken. The contextinformation is desirably automatically learned and stored with thepicture in the social media account of Susan, using the learning methodsdescribed herein. Alternatively, the Eiffel Tower photo can be part ofthe sister's social media account, and because her sister is a communitymember and has given sharing rights, the social media server computerautomatically shares the photo to Susan's digital picture frame.Desirably, any photo relevant to Susan can be automatically shared fromany of Susan's community members and automatically shown on the pictureframe display.

In one embodiment of this invention, the digital picture frame accessessocial media or her electronic device without detailed instructions fromSusan and loads and sequences photos displayed on the digital display asa function of profiling traits selected from chronological order, photolocation, photo activity, and/or community member. Having automaticallylearned the location, activity, and community member involvement on aper photo or video basis using the method described herein, a “storytelling” capability is supported. That is, in storytelling,chronological stories, optionally simultaneously displayed on a splitscreen, are grouped by: purely time, namely in sequential chronologicalordering; location, namely a traversal of sites on a location basedtrip; activity, namely in chronological ordering of a given or similarset of activities; community member involvement, namely a pictorialinteraction with community members, potentially segmented by particularcommunity member or members; or any other profiling trait(s) of arecognized user that can be used to cluster or segment photos forautomatic story telling.

In one embodiment of this invention, Susan's friend Mary joins Susan inviewing the picture frame, either directly or within a detectablevicinity of the picture frame. The picture frame automatically detectsthe additional presence of Mary, automatically detecting her identity,automatically obtains or loads Mary's profile, and automatically changesthe photos displayed on the digital display to those relevant to bothSusan and Mary. The photos could be obtained from Susan's social mediaaccount(s) or personal picture capture or storage electronic device butpreferably, the photos are obtained from both Susan's and Mary's socialmedia accounts and/or local personal picture capture or storageelectronic devices, e.g., mobile phones 29 and 29′. Desirably the photosfrom each of the two viewers are shared activity photos; from activitiesshared by the two friends. The frame 20 and/or server computer 26automatically determines shared activity photos of the two persons as afunction of the learned context information automatically associatedwith the shared activity photos by the server computer. As more peopleare automatically detected, photos from more accounts or devices can beautomatically added, and the photos automatically organized by contextsuch as photo location, a photo activity, and the present communitymembers. Options can be added on the social media accounts to share ornot share photos with community members or their devices according tothis invention.

In another embodiment, Mary and Susan each simultaneously, or withinsome specified duration of time, look at their respective digitalpicture frame 20 and 20′, such as linked as shown in FIG. 1 or networkconnected when each frame in a different location (e.g., theirrespective residences). Remote frames can communicate, via a network,such as through server 26, and indicate to both Mary and Susan, each attheir respective frames 20 and 20′, their respective presence,potentially indicating when Mary and Susan are both viewing or viewed aphotograph from a preselected set of photographs within the specifiedduration of time. The duration of time can be defined in terms of aspecific time or as within a time interval commencing from when eitherMary or Susan viewed the given photograph from within the preselectedset. The frames 20 and 20′ can automatically or upon instructioncoordinate so that the two are viewing the same photos. Likewise thephotos can be augmented to included shared experiences, and be uploadedfrom both of Mary and Susan's linked devices 29 and 29′ and/orrespective social media account.

In embodiments of this invention, photographs from digital photocollections are provided to the frame in multiple ways including but notlimited to self-capture, transmission from remote photograph capturingdevices, from other frames, and through server transmission. The frameeither receives photographs taken by other devices or itself takes aphotograph, if so configured. The digital photo collection(s) from whichphotos can be obtained and sent to the frame can be associated with oneor more additional viewers and/or digital picture frames, and/or anyelectronic device of a community member of the viewer. This allows forsending new photographs, such as from a mobile device app, to acommunity member frame (e.g. sending grandmother's frame today's T-ballpictures).

Independent of the means by which a photograph is obtained, at times,the main artifact within the photograph, such as, for example, the faceof a person in the picture, is not properly aligned. As such, whendisplayed on the frame, either the main artifact is misaligned, orpotentially, is removed all together. To mitigate potentialmisalignment, the frame and method of this invention can include orprovide an automated smart cropping feature for automatically croppingand aligning the photos to fit the digital display of the frame.

For proper viewing, the frame or the server driving the disclosed framecan automatically crop and/or realign images. Without requiring userinteraction, the server executes cropping software that realigns animage for proper display on the frame to ensure that the main artifactwithin the image is automatically scaled and realigned to a given aspectratio (4:3 or 3:4 for example). The artifact may not be a single thingor entity, and, for example, could be a group such as two faces in theforeground close to each other or a tight group of people in a familyshot. An exemplary non-limiting approach identifies the face areas andattempts to align a corresponding area at either center to top-33% ofthe image to be portrayed, depending on the size. Large faces can becentered vertically whereas smaller faces can be aligned in a top 33% ofthe display to show more of the person's body. Thus, when presented, theimage is esoterically properly displayed.

Embodiments of this invention preferably also include image filtering,either at the frame or server level. Filtering can include removingimages of faulty artifacts, such as, for example, low quality (blurry,etc.) images. Artifacts of unlikely interest can also be filtered. Forexample, photographs of pets containing faces are typically of interest,however animal photographs without heads are typically not of interest,and can be removed. The curator of the photographs, such as the owner orviewer of the frame, can additionally impose other image or contentquality constraints, Content constraints can span people, places,artifacts, and actions; namely anything that can be recognizedautomatically by any image, photograph, or document processingtechniques known in the art.

Embodiments of this invention enable the owner or user of a frame tospecify additional constraints on the photographs displayed. Such as inthe case of the photograph curator, the frame owner too may designatecontent restrictions; for example, pornographic photographs can beprohibited to display. Such constraining rules are defined andincorporated into the filtering approach. Note that it is also withinthe scope of this invention that the constraining rules include but arenot limited to lighting characteristics, scene selections, number ofindividuals displayed, distance or horizon specifications, or any otherphotograph characterization features.

The invention further includes receiver (displayed) side based filteringwhere the filtering of content to display to be defined by not only bythe curator/owner of the photographs but also the owner/user of thedisplaying frame. In the case of multiple owners/users of a displayingframe, the identity of the current viewer can be used to select thefiltering limitations imposed. It is also within the scope of thisinvention that a default set of filtering limitations are imposed shouldthe viewer not be identified, multiple viewers be detected, or any otherpre-specified set of conditions. Constraints imposed on the quality ofthe photograph to be displayed can be implemented, but also constraintsthat filter based on the sender's (photograph capturer's) desires;namely the sender imposed restrictions on the photographs to bedisplayed on the receiving frame. The receiver, namely the frame'sowner/user, also has the ability to limit the photographs to displaybased on their image quality or on their content.

The frame's owner/user provides the sender access to the frame. Thisaccess, without any receiver side filtering, aside from image qualityconstraints, gives the sender control over the content to be sent andcorrespondingly displayed on the receiving frame. By allowing thereceiver to limit those photographs that are to be displayed, a greatersense of comfort and security are provided to the frame's owner; hence,the frame's owner is more likely to allow additional senders to accessthe frame, and thus, the community associated with the given frame islikely to grow. Receiver-side filtering rules can include, but are notlimited to: general content restrictions, such as but not limited torestricting the display of nude photographs or photographs where toomany people are included; individual specific restrictions, such as butnot limited to restricting any photographs that include a previousspouse; or restrictions relying on metadata constraints, such as, butnot limited to, restricting photographs taken prior to a certain date ortaken at a certain location.

FIG. 2 illustrates an embodiment of the invention includingreceiver-side photo filtering. In FIG. 2, Susan is at home with herdigital picture frame 20. Susan's daughter Sally and Sally's husband Bobare in a remote location, where Bob takes a picture of Sally on Bob'smobile device 29. Bob is a community member with Susan (and both withSally), and Susan has set content constraints to allow pictures of Sallytaken by any community member to be displayed on her frame 20. Inembodiments of this invention, when Bob takes the picture of Sally, thepicture of Sally is automatically identified on device 29 and routedover a network, such as via social media server computer 26, to Susan'sframe 20 for display.

To route the picture to Susan, the content of the digital photo isautomatically identified, and any target electronic device, such asSusan's frame 20, that would receive the photo also needs to beidentified and matched to the digital photo. The identified content isautomatically compared to content constraints of Susan's frame 20 todetermine if the digital photo should be routed to Susan's frame 20.Likewise other community member devices 29′ can be similarly processedeach time a photo is taken thereon to determine which, if any, photomatches or meets Susan's content constraints and should be routed anddisplayed on Susan's frame 20. Thus Susan can automatically receive, forexample, photos of each family member automatically upon the photosbeing taken. Susan can control the content and quality of the photos, toexclude, as examples, photos taken by community members of or includingnon-community members, poorly ‘angled’ selfies, photos taken with orthrough particular apps (e.g., Snapchat®), fuzzy, grainy or poorly litphotos, or duplicate photos (e.g., photos taken within a predeterminedtimeframe of each other). Photos not meeting Susan's content constraintsare desirably not routed to Susan's frame 20′.

In embodiments of this invention, content identification and/or arouting decision is/are determined by a content restriction filter. Thecontent restriction filter can be implemented via software coding storedand executed on, referring to FIG. 2, the device 29, server 26, and/orframe 20. In embodiments of this invention, the filter components can beimplemented in modules on more than one device. For example, a photoquality filter module can be implemented on the photo-taking device 29,to limit photos sent to the server 26. The server can then performcontent identification on photos received, and route each photo to thetarget frame for which the photo meets the content constraints. Photoprocessing, such as cropping, aligning, scaling, enhancement, andrepetition filtering can be performed on the frame, server, and/ormobile device, as needed.

The digital photo frame 20 can additionally detect via the cameraSusan's physical response, for example a body movement or facialexpression, to the routed/displayed photo, and an interest level gradebased on the physical response can be associated with the photo. Theinterest level grade can be used to determine whether the digital photois replayed, or a frequency at which it is replayed. The interest levelgrade can additionally be used to automatically update the contentconstraints, such as if Susan indicates a low interest to one or morephotos of Sally with her tongue out, or with Sally at a particularlocation (e.g., a bar). In embodiments of this invention, the frame candisplay a message or message indicator requesting the physical response,for example so the frame can learn what photos to display.

FIG. 2 further illustrates a second digital photo frame 20′ of Susan'sfriend Mary, remote from the frame 20. In an exemplary embodiment, Maryis a community member with Susan, Sally, and Bob, but Mary has notadded/allowed photos of Sally or Bob in the content constraints of frame20′. However, Mary loves Paris, and wishes to see all photos related toParis from her community members. If Bob's picture of Mary with mobiledevice 29 occurs in front of the Eiffel Tower, then the contentrestriction filter module(s) will allow the photo to route to Susan'sframe 20 because Sally is present, and will also route the digital phototo Mary's frame 20′ for including content related to Paris. In this way,community members can learn of activities of other community membersthat are of particular interest, without receiving every photo from theother community members.

The invention includes various standalone picture frames. FIG. 3 shows arear view of a picture frame 120 according to one embodiment of thisinvention. The picture frame includes a housing 122 with a display (notshown) on a first, front side, and a back panel 124 on an oppositesecond side from the display. The back panel 124 includes anasymmetrical nonplanar shape. As shown, the asymmetrical nonplanar shapeis an oblique pyramidal shape extending outward, or away from thedisplay side. The back panel 124 includes four irregular trapezoidalsections 126, each having a different size and shape. The four irregulartrapezoidal sections 126 surround a stand attachment 130.

The stand attachment 130, and the corresponding stand 132 when attached,are off-center on the back panel 124. As shown, the stand attachment 130is centered between the four irregular trapezoidal sections 126. Thestand 132 is a rotating stand, which rotates to allow the frame 120 tostand in the portrait position or the landscape position by rotating 90degrees between two positions. The stand 132 includes a circular element134 that fits, such as by a snap fit, within a circular recess 136 ofthe stand attachment 130, to form a rotating joint. In embodiments ofthis invention, a stop mechanism can be used to limit a rotation of thestand 130 to 90 degrees. As shown, the stand attachment 130 includes arotation limiter 140 that, in combination with a corresponding limiterelement 142 on the stand 132, does not allow the stand 132 to turn morethan necessary for the desired two positions.

The frame 120 includes a power port 150 for a corresponding power cord.The power port 150 is desirably within the stand attachment 130, and canbe covered by the stand 132. The power port 150 and correspondingconnector on the power cord are desirably each circular to allow theconnector to rotate within the power port when the stand 132 is rotated.

FIGS. 4A-D show a frame 120 and a rotating stand 132 according to oneembodiment of this invention. In FIG. 4A, a power cord 155, having a90-degree-angled connector 156, is attached to a power port in the standattachment 130. The stand 132 is attached over the power cord connector156, as shown in FIG. 4B. FIG. 4B shows a horizontal or landscapeposition. FIGS. 4C-D show the rotating stand 132 moving to allow theframe 120 to sit in the vertical or portrait position of FIG. 4D. Thestand 132 rotates to sit the frame 120 upon either of a pair ofperpendicularly abutting frame sides, without the need to remove thestand 132. The offset position of the stand allows the stand to rotateand support the frame with the desired lean in both positions. Varioussizes, shapes, and configurations are available for the back panel,stand, and stand attachment of this invention, depending on need.

FIG. 5 shows a picture frame 120 according to another embodiment of thisinvention. The back panel 124 including an off-center rotatable stand132, likewise over a power cord 155. The picture frame 120 includes ahollow mesh edge section 121 extending about the display and a centerportion of the back panel 124. The frame 120 includes two touch bars170, each disposed on a pair of perpendicularly abutting frame sides.The touch bars 170 can control a frame feature, such as being a slidethat allows a user to manually change a digital photo on the display. Bysliding a finger along the touch bar 170, the displayed photo islikewise swiped from, or returned to, the display. The touch bars canalso be used for volume or brightness settings. The touch bars 170 areon the two sides opposite the two sides that the frame would sit on inthe portrait and landscape positions, respectively, such that one touchbar is always on the top side of the frame.

The stand attachment of FIG. 5 can be a rotatable joint as discussedabove. Alternatively a slide mechanism, such as shown in FIG. 6 can beused to allow the stand 132 to be removed from the stand attachment 130and repositioned in the other 90 degree position. The stand attachment132 an be an extension 134 with an upper ridge 136 that extends aroundthe extension 134. The stand 132 can include a corresponding groove 138to receive the ridge 136, in either of the two desired stand positions.The user can slide the stand 132 off the attachment 130 and slide thestand 132 back on with a 90 degree change of position. Other standattachment configurations that allow a 90 degree change of position canalso be used, depending on need.

FIGS. 7A-B shows a frame 120 according to another embodiment of theinvention. The frame 120 includes a wide edge 125 about the frameperiphery that allows the frame 120 to be upright in both positionswithout a separate stand. The back panel 124 includes an asymmetricalnonplanar shape and an off-center power port 150. The back panelincludes an oblique shape, of four irregular trapezoidal sections 126,that is recessed into the housing. The power port 150 is within theoblique pyramidal shape, centered between the four irregular trapezoidalsections. The frame 120 further includes two touch bars 170, with one ofthe touch bars 170 on top in the landscape position of FIG. 7A, and theother touch bar 170 on top in the portrait position of FIG. 7B.

A non-limiting embodiment of this invention consists of at least onepicture capture or storage electronic device 29 communicating directlyvia a network or wired or other wireless connection with the digitalpicture frame 20. To support such interaction, the picture capture orstorage device 29 connects via the network to the digital picture frame20 and downloads the digital picture frame interaction software to thedevice 29. Once installed, the application software scans all theavailable pictures on the device. The remaining process description isillustrated using a non-limiting example based on face selection. It islikewise, however, within the scope of this invention to select based onobjects other than faces including but not limited to scenes, materials,and activities. A second device 29′ (or more) can likewise link to theframe upon being in connection proximity.

Once selected, all photos containing faces are clustered by theapplication, either at the server or frame level. Individual photographscan then be automatically filtered based on features. Some of thesefeatures are quality related, e.g., red-eyes or blurred; some featuresare content related, e.g., excluding certain activities, people, orlocations. Those photographs that remain selected can be tagged withadditional metadata. Metadata includes all photograph generatedinformation, such as but not limited to location, time, or weather, aswell as derived data, such as but not limited to cluster identity.

The invention can use clustering to form grouped clusters according tophoto content. The clustering is desirably performed as a function of acommon detected content in the photos, such as things, places,activities, or combinations thereof. The clustering of photos can beused for determining a photo slideshow on the digital display of thephotos from a digital photo collection as a function of the clustering.The grouped cluster can be integrated to a slideshow as preferredimages, or used to provide requested slideshows, such as “show grandma”.

Embodiments of this invention include identity clustering, such as bylongitudinal facial clustering. Photographs are clustered not only inaccordance to an individual identity at a given time, namely at a givenage, but the disclosed longitudinal facial clustering clustersindividuals throughout their lifetime. Consider a collection of picturesthat span a prolonged period of time. In such a case, an individualnaturally ages, and creating a collage of that particular individualnecessitates accounting for their aging process. The disclosed approachuses photograph metadata, including but not limited to the date thephotograph was taken, to age the individuals to a common age using anyof the individual aging techniques known in the art. Once at a commonage, clustering is performed to group pictures of the same individual.

FIGS. 8 and 9 illustrate a clustering of photos of Queen Elizabeth. Inone cluster (“Young Cluster”) are photographs of the young Queen. Herage progression is represented in various sub-clusters forming a photochain to a second cluster (“Old Cluster”). Note however that thesephotographs are all of Queen Elizabeth, and as such, they are alleventually clustered together (as represented by the linking edges) toform a larger cluster. The sub-clusters can be linked together byfeatures that do not change as much over time, such as eye distance,head shape, nose shape, etc. Clusters according to this invention can beconstantly updated and adjusted, such as using new photos and/or othernew metadata.

An alternative or additional clustering technique uses statisticalmodels of metadata. Using metadata, such as, but not limited to,location identification or date and time, the likelihood of a givenperson being at a given location at a given time is computed. Shouldphotographs support such a given likelihood, they too are potentiallyclustered.

Similarly, metadata can be used to prevent the clustering of an unlikelyset of photographs. For a non-limiting example, image-processingtechniques are somewhat inaccurate at recognizing and disambiguatingchildren. This can be particularly true for infants and young children.Photo metadata that indicate when and where a photograph was taken canbe used to reduce or eliminate disambiguation. For example, photographsof children taken at a close time interval but at vastly separatedgeographical locations are likely to be different individuals.Similarly, children of a similar age but appearing in photographs takenfar apart in time are again likely to be different individuals.

Clustering in additional embodiments of this invention employs a logicaltwo-phase approach. It is within the scope of this invention that the“two phases” are integrated into a single phase, or are partitioned overa greater number of phases. Initially, a cluster is formed using facialrecognition using any one or more of the many known in the artclustering techniques. This phase clusters photographs of individualsthat closely resemble across all photographs. However, should anindividual modify their appearance either intentionally, say but notlimited to using a costume, or unintentionally, say but not limited to alopsided profile, an erroneous separate cluster may result. Independentof cause, the merging of clusters of the same individual is needed. Thesecond phase uses known image processing techniques to extract featuresof photographs within the given cluster. These features are used torepresent the cluster, say via a representative centroid. By averagingthe features across the photographs within the cluster, the effects ofminor variances in some of the features are diminished. The centroids ofall clusters are compared. If centroids of differing clusters aresufficiently close, the photographs within the close clusters arefurther examined. This further examination includes but is not limitedto the use of respective metadata. If still deemed similar, the clustersare merged.

In another embodiment, the disclosed photograph clustering approachcombines clustering with classification. Current clusters form the basisfor inducing classification models. These models, in turn, are used toassign currently unassigned photographs to existing clusters if thesimilarity measure score employed meets or exceeds the discriminativethreshold. It is within the scope of this invention to employ any of themany known in the art image similarity measures.

In another embodiment, the linking of similar photographs across framesor digital photo collections is performed. Frame and/or mobile devicephotograph collections are maintained separately, however the submissionof pictures to a frame can constitute a community membership. Withinsuch a community, photographs of individuals portrayed might be common.In some embodiments of this invention clusters of photographs acrossframes/collections are compared and the server suggests to the owners ofthe frames, as well as to those submitting pictures to the frames, tosend photographs of common individuals to frames that already possessphotographs of those individuals, but not necessarily the identifiedphotographs. Thus, photograph sharing is fostered. It is also within thescope of this invention to impose a threshold on the minimum number ofphotographs that a particular individual must appear in a remote frameor other digital photo collection prior to suggesting that additionalpictures of that individual be routed to that remote frame.

Another embodiment clusters photographs based on common detectedobjects. Exemplary objects include artifacts (things, places,activities) and living beings (typically people and pets). Locations andactivities can be detected via metadata and individuals present can bedetected via facial recognition.

Photo data can be preprocessed on the photograph-capturing device. Theclustering and cropping approaches described herein assume theavailability of the data on a server. However, given current computingcapabilities on the photograph capturing devices, such as but notlimited to GPUs, significant preprocessing can immediately occur on theactual device. In doing so, the bandwidth required to transfer thephotographs to the server is vastly reduced. In embodiments of thisinvention, preprocessing includes, but is not limited to: image qualitydetection and enhancement (if photographs remain of poor quality thenexclude); content filtering (for example, pornographic or choppedfeature images are excluded); and repetitive (same picture takenmultiple times, only highest of quality is sent).

Additionally, picture fingerprints (compressed or very low resolutionversions) can be generated on the capturing device, and only thefingerprints are initially sent to the server. If the server deems thesepictures as desired, based on any pre-established criteria, the serverinforms the device, and the original pictures are transferred. Usingsuch an approach, only desired pictures are sent to the server, againreducing the bandwidth involved in the transfer.

Cross capturing-device filtering can also be incorporated. Using picturefingerprints, devices can interact directly among themselves todetermine which photographs to transfer to the server. For example, aset of community members may wish to strictly share images of commoninterest, and in embodiments of this invention the devices can sharefingerprints and automatically filter photographs of similar nature.Thus, only one of the photographs, potentially evaluated to be of thehighest of quality among the similar pictures, is transferred to theserver, again reducing the transfer bandwidth. Additionally, a votingprocedure can be imposed where only those picture with a sufficientlyhigh vote are transferred.

Likewise within the scope of this invention are other criteria by whichto select the photograph to transfer. Non-limiting additional oralternative criteria include but are not limited to the poweravailability or bandwidth capacity of the capturing device. As anexample, if a device has low bandwidth capacity or is nearly out ofpower, another device might be selected to send its version of a similarphotograph.

Photographs are posted to frames for the pleasure of the receiver,namely a viewer of the frame. In embodiments of this invention, theframe provides the frame viewer with the capability to provide feedbackto the sender of the photograph. Via a physical response, such as a handgesture, the viewer can send, if they so wish, an indication of thedegree of excitement or dissatisfaction with a given photograph. Oncethe gesture is captured and interpreted, the frame sends the server amessage indicating the viewer's interest level in the photograph, whichin turn, is stored and applied at the server level and/or indicated tothe sender of the photograph. Such a feedback loop motivates the sendingdevice or collection to automatically send or not to send similaradditional photographs. Likewise using facial recognition, the frame canidentify the viewer, and if desired, tag the feedback with the viewer'sidentity. Also the frame can infer feedback expressed via facialexpressions using facial recognition. Similarly, other body movementgestures, such as but not limited to the nodding or shaking of headrepresenting positive or negative feedback, respectively, can becaptured and treated as feedback.

The invention further includes ranking the photos to be circulated onthe frame, such as for the purpose of automatically determining an orderof presentation, a number of display repetitions, and/or a time periodof display for each of the photos as a function of the ranking. For anygiven frame, the photographs sent to the frame are circulated. Theordering of presentation of photographs for a given frame can be basedon a ranking. In one ranking, similar photographs are clustered basedon, for example, but not limited to people, places, things, oractivities. Photographs within these clusters are then shown insuccession or within a predefined, or viewer set, separation of eachother. In another ranking application, feedback provided by the frameviewer(s) is used to determine the ordering, with bias given towards themore favorable photographs. In yet another ranking application,photographs are presented in a random order. In yet another rankingapplication, popular photographs, based on feedback, are repeated,possibly even frequently. In yet another ranking application, allphotographs are shown prior to any repetition. In yet another rankingapplication, the time stamp of the photograph is alternatively oradditionally used for ranking. That is, a chronologically ascending ordescending order of photographs can be presented. In yet another rankingapplication, photographs are displayed according to similar seasons ordates within successive or separated years. In yet another rankingapplication, a popular photograph is displayed longer prior to beingreplaced by its successor. In yet another ranking application, theduration of display is the same for all photographs. In yet anotherranking application, photographs are clustered according to time slices,the number of photographs within each time slice is computed, andphotographs within popular time slices are shown more frequently. It iswithin the scope of this invention to vary the time slicessignificantly, whereby brief time slices with high photograph countstend to indicate great interest in the shown event, location, orindividuals. Lengthy time slices, without loss of generality, canaccount for trips, seasons, or any sustained activity or event. In yetanother ranking application, the viewer explicitly specifies theordering and duration of display. One skilled in the art recognizes thatany ranking rule known in the art can be imposed and that thesepresented rankings serve only as non-limiting examples. More so,multiple combinations of these and other ranking schemes are likewisesupported by this invention.

Another embodiment of this invention includes, and the digital pictureframe is implemented with, a method, system, and/or apparatus, such asembodied in an MPSM or other software application, that automaticallydetermines and shares a location, an activity, and/or photos of a user.The application learns user activity over time, with the learning basedupon user locations and/or context. The application can learn throughautomatically determining activities at locations based upon knowncontext information and past context information for the location. Theapplication can tag photos for determining context relevancy for showingon the digital picture frame, as discussed above. The invention furtherincludes energy saving location methods for the mobile device that canbe used to more efficiently allow the location and social media aspectsof the invention to be implemented on a mobile device. The method andapplication can be used for any suitable function, such as a safetyand/or reminder serves, and is particularly useful for use in socialmedia applications and for generating photos for display on the digitalpicture frame. The invention will be described below with implementationin an MPSM system, and particularly with an MPSM application that learnsuser activity over time, with the learning based upon user locationsand/or context.

The MPSM method and system of this invention is mobile and positional innature. Such systems, like many other systems originally developed onone type of computing platform but migrated to another, operate not onlyon mobile environments. That is, while MPSM implementations are targetedto primarily execute on mobile devices, such as but not limited tosmart-phones, tablets, and/or laptops, they often support implementationfor non-mobile environments such as but not limited to desktops andworkstations, servers, and large scale compute farms and cloud computingservers. The invention will be described below with a mobile device,such as smart phone having cell service, a GPS system, and access to theInternet via WiFi.

The MPSM method and system of this invention is desirably executed orimplemented on and/or through a mobile device computing platform. Suchcomputing platforms generally include a processor, a recordable medium,an input/output (I/O) device, and a network interface capable ofconnecting either directly or indirectly to the Internet. The mobiledevice executes over a networked environment, a non-limiting exampleshown in FIG. 10. The mobile device is connected, either directly orindirectly, using any of the many techniques and technologies known inthe art, over a network, to back-end system or systems,itself/themselves computing devices. The mobile device can connect witha remote server, shown in FIG. 10 as server 38, to store and/or accessuser or community information.

MPSM systems are used to support users remaining socially aware of theircommunity. That is, their primary usage typically is to actively monitorthe location and activity of family members, friends, colleagues, andgenerally others within one's community. Communities can be partitionedinto sub-communities where the union of the sub-communities forms theuser's community. The sub-communities may or may not overlap. Thepartitioning of communities into sub-communities is beneficial insupporting specialized applications. For example, while a user mighthave general interest in the location and activity of all of theircommunity members, they might be particularly interested in the locationand activity of those who might be suddenly in need of assistance.

The creation of a community can include the issuing of invitations. Aninvitation is a request by a user A of another user B to allow theinviting user, user A, to track the activities of the invited user, userB, and vice versa. If the invited user accepts, the inviting and invitedusers form a community.

A community is relevant to only that user which formed it. That is,different users have different communities. A community is a grouping ofinvited (referred to as remote) users by the inviting (referred to aslocal) user. A local user can partition or merge a community, thusforming a sub-community or a parent community, respectively. Forexample, consider 5 users: Bob, Sam, Sally, Alice, and Susan. Bob caninvite Sam, Sally, and Alice, thus forming his user community. Bob canlikewise partition his community into a sub-community consisting of onlySam and Sally. Sally can invite Susan. Thus, Sally's community wouldinclude Bob (via his invitation) as well as Susan. If no additionalinvites occurred, Sam's and Alice's respective communities would onlyinclude Bob (each via Bob's invitation), while Susan's community wouldonly include Sally (via Sally's invitation).

Providing users with the opportunity to expand their communities in aconvenient manner is advantageous. Such expansion can seamlessly beaccommodated by including users listed in a user's contact lists eitheras a whole or selectively into their community. Contact lists include,but are not limited to, users listed in a user's local address book,e-mail contact list, Twitter follow list, LinkedIn connections list,and/or Facebook friends list. By incorporating users listed in a user'scontact list, the user's community is expanded without effort. Note,however, that selected inclusion can be supported; thus enablingcommunity growth without unnecessarily over-expanding the community.That is, entries from the contact list can be included in their entiretyand the user can selectively remove those entries which s/he wishes tobe excluded from the community. Similarly, entries from the contact listcan be selectively added.

Users are identified by their account identifier. To use MPSM a useraccount is created. User accounts generally require a user login, whichis a unique user identifier, and a password or equivalent. After havingcreated an account, a user can log in. Initially, the local user doesnot have a community. In embodiments of this invention, over time, themethod and application tracks the activities and location of the localuser. Should the local user establish a community as aforementioneddescribed, the community members will likewise be tracked. Local usersreceive notifications of the location and activities of their communitymembers. Once logged in, the local user can select to activate ordeactivate self and community tracking and notification. If notoverwritten, default settings are used.

Whenever logged in and tracking is enabled, a user's location andactivity is tracked. That is, a user periodically records their locationand/or activity. Locations are tagged by name. Names can be but are notlimited to the following schemes: physical (e.g., 123 Oak St.), absolute(e.g., Acme Coffee), and/or relative (e.g., my work office), orproximity (e.g., two miles from home). Activities are typically events.These events might be common to the entire community such as: “drinkingcoffee,” “eating lunch,” “sampling wine,” “working from home,”“commuting,” etc., to more specific to a local user such as “restoringcar” or “driving to lake home.” Multiple activities can occursimultaneously. Users can change their activities at any time.

Unless preloaded or derived from an external source, such as but notlimited to a location database, initially, all locations and activitiesare unknown. Local users must record all such location-activitycombinations, i.e., a local user must name or tag the location and theassociated activity. A list of activities common to the local user'scommunity can be provided. This community activity list can be rankedeither arbitrarily (randomly), according to most recently used, mostfrequently used, relevance to location, alphabetically, etc. Eventually,an activity list specific to the local user is learned. This local useractivity list can be displayed to the local user either individually,along with the community list, or merged with the community list. Again,any of these lists can be ranked as previously mentioned.

FIG. 10 illustrates a representative area 30 to demonstrate a method ofand application for locations and/or activities of a user participatingin a social networking service. The area 30 is shown as a cellularcommunication network including a plurality of cells 32 each disposedaround a cellular communication antennae or base station 36. Within thearea are a plurality of destinations each shown as including a WiFiInternet connection. The local user has one or more electronic devices,such as a mobile device that communicates with a remote server 38 viathe cellular network and/or the WiFi connections. As will be appreciatedthe methods and applications of this invention can operate within anysuitable size and configuration of the communication area, depending onwhat the user encounters.

Destination 40 is the home of the user. The user commutes to office 42for work on most business days. On the way the user typically stops atthe coffee shop 41. For lunch on most days, the user visits restaurant43, but on Wednesdays the user typically meets a second user for lunchat restaurant 44.

At each destination 40-44, the user enters user information about thedestination. The application and computer system that receives the userinformation automatically associates the user information with thedestination, and stores the user information in a locations database,such as on the device and/or at server 38. The destination desirably isdetermined automatically and tagged with the user information, such as alocation name of the destination and/or the user activity beingperformed at the destination. For example, destination 40 can be taggedas “home” and likely has numerous activities associated with it. Thedestination 41, and any photos taken, can be tagged as its establishmentname “Acme Coffee” or simply “coffee shop” and associated with the useractivity of “buying coffee” or “latte time.” The manually entered userinformation can then be automatically shared to the user's community ina social networking service. Similar user information is received forthe other destinations 42-44. The user information desirably includesany other information about the location or activity, whether manuallyentered or automatically determined, such as the time of the visit oractivity. Some destinations, such as home or work will likely havemultiple user activities over a period of time, such as “coffee break,”“meeting time,” and/or “quitting time.”

The computer system receives user information and associates the userinformation with the corresponding destination, and any photos taken,for multiple visits to each of the destinations 40-44. The computersystem begins learning the locations and user activities. In embodimentsof this invention, the user can be automatically prompted forconfirmation of the user information upon arriving at a destination toconfirm the location and/or user activity. For example, the user can beprovided with an automatically generated list of previously entered useractivities for the destination upon arrival, thereby promoting efficientcollection of information. The items on the list can be listed in anorder based upon a particular ranking, such as number of times enteredpreviously or based upon a context, such as what activity is likelybeing performed at a particular time of a particular day.

Over time, the computer system learns the user information and beginsautomatically associating and identifying at least some user activitiesfor corresponding locations and any photos taken. As will beappreciated, the automatic identifying of activities at locations willlikely occur at different rates for different activities and locations,with some locations having fewer activities and/or more frequent visitsthan others. In preferred embodiments of this invention, the systemautomatically shares the user information in a social networking serviceupon automatically detecting further user arrivals at the destination.Photos taken are likewise automatically tagged with the userinformation. The automatic sharing of user locations and/or activitiesdesirably occurs upon the user's arrival at the location, or at aparticular time at the location. As such the invention includes anautomatic detection of the user's arrival at a destination. Theautomatic sharing and photo tagging desirably operates without useraction and prior to receiving any additional user information for thedestination.

As an example, the user may typically purchase lunch at destination 43,but on Wednesdays typically goes to lunch with a friend or spouse atdestination 44. The lunch routines of the user, and particularly theWednesday lunch routine, can be learned by the system and automaticallyshared to the user's community upon the system automatically determiningarrival, without manually input from the user. If the user is havinglunch with a community member, then the system can automaticallydetermine that both users are at the same location together toautomatically recognize and confirm the lunch activity, and proceed toautomatically share the information for both user's to their respectivecommunities. If the user deviates from a routine, the system can knowthis, and refrain from sharing the typical destination, by the mobiledevice detecting a different location than the typical routinedestination.

In embodiments of this invention, learning is accomplished by any knownmachine learning, data mining, and/or statistical techniques known inthe art. Supervised, semi-supervised, and/or unsupervised approaches canbe used, including, but not limited to Naïve Bayes, Neural Networks,Support Vector Machine, and/or Associating Mining based techniques.

The MPSM method and system of this invention desirably records allposted locations and activities. Throughout use, the disclosed inventionlearns the corresponding locations and the set of associated activities.More so, via comments made by the local user and by the local user'scommunities, the importance of the activities can be learned, such asfor the prompting discussed above. Importance can be either local useror community biased. Additionally, importance can be biased by context.For example, community members as a whole might prefer “eating steak,”“eating pizza,” and “eating sushi,” in that respective order. On theother hand, a local user might only eat sushi. Thus, local user biaswill yield “eating sushi” only, while community bias will suggest“eating steak,” “eating pizza,” and “eating sushi,” in that respectiveorder.

In embodiments of the MPSM method and system of this invention,locations are named according to a naming convention. Regardless of thenaming convention used, a location is a physical geographical position.More so, physical geographic locations associate properties that canvary with or be dependent on context, namely time and date (hours, dayof week, calendar date, etc.), users involved, and their relationshipsto each other, etc. This context can affect the associated location nameor activity.

A common scheme that can be used to at least assist in identifying aphysical geographical location is via the use of geocoding. Geocoding isthe representation of a physical location via the pairing of latitudinaland longitudinal coordinates commonly referred to as a lat-long pair.Global Positioning Systems (GPS) can also determine a physical positioncoordinated via the triangulation of satellite transmissions. TypicallyGPS devices derive lat-long pairs which are made available to a varietyof applications, often via map displays. GPS economics, accuracy, andsimplicity of use resulted in their wide appeal and commercial success.Their continuous use in mobile devices is problematic, however, as theyare energy intensive and rapidly drain the battery. Thus, alternativemeans or approaches to detect locations are desired.

Embodiments of the MPSM method and system of this invention, asdiscussed above in FIG. 10, use or rely upon cell coordinates. Whenmobile devices communicate with a cell tower, they send their cellcoordinates. These coordinates are recorded by the cell provider and aretypically not publicly known. The cell phone or, in this case, themobile device supporting the positional social media system, however, isaware of their coordinates. Thus, the device can store the cellcoordinate position and automatically associate that cell coordinatewith the location name provided by the local user. Over time, a locationdatabase of cell coordinate and named location pairs is created. Thelocal portion of the database favors the local user. The union of allthe local portions of the location database desirably constitutes thename space of the entire MPSM system of this invention. It is understoodthat any of the many database management systems or storage schemesknown in the art can serve as the platform for this location database.Thus, location names can be provided without the need to rely on aglobal positioning system, reducing battery consumption. Location datacan additionally or alternatively be purchased or otherwise provided bya third party.

An additional and/or alternative approach for automatic locationdetermination relies on WiFi triangulations. Mobile devices can grow andmaintain a database of known open WiFi networks, for clarity we callthis database an Open-WiFi-Net database. Such mobile devices can use theinformation stored or derived from the information stored in theOpen-WiFi-Net database to further refine the accuracy of a locationwithout the use of GPS. Via point triangulation, when an Open-WiFi-Netdatabase is available, the mobile operating system uses not only thecell tower but also WiFi triangulations to determine location. It iswithin the scope of this invention to use either or both cell-phone andWiFi triangulations to enhance location information in addition to anyother disclosed approach. The mobile device can use the WiFi signal at adestination, such as destination 43, and additionally or alternativelyany detectable open WiFi signal from a neighboring location, such asestablishment 45 that is adjacent destination 43.

Having created the location database, searching, namely querying, thedatabase uses the cell coordinate or the location name. That is, alocation name query takes a location name as input and returns thecorresponding cell coordinate. A cell coordinate query takes a locationname as input and returns the corresponding location name. Note that,multiple names can be attributed to a given cell coordinate. That is, alocal user might name a location using multiple different names;different users can name same locations using different names.Similarly, the same name can be used for different cell coordinatelocations. All names corresponding to a given cell coordinate arereturned. It is within the scope of this invention to selectively returnnames based on context, user, or community bias. Similarly, all cellcoordinates corresponding to a given name are returned. Again, it iswithin the scope of this invention to selectively return coordinatesbased on context, user, or community bias. Ranking of the resultsreturned can, when desired, be biased towards the local user.

A key concern for MPSM systems is collecting location information.Clearly any location information available within the mobile deviceshould be harnessed. Thus, if GPS readings or any other locationinformation is generated by other device resident applications, thesereadings are desirably recorded and utilized by the method andapplication of this invention. However, reliance on strictly otherapplications to obtain positional information is obviously not realisticor possible.

In embodiments of the MPSM method and system of this invention,positional information is obtained via the use of geofences. A geofenceis geographical boundary or “fence” surrounding a positional reading. Asthese boundaries are radius based, geofences are generally circular.Location transmission occurs whenever a handover of one cell tower toanother occurs and is expected but not guaranteed to occur once ageofence boundary is crossed. To track location, periodic locationtransmissions are required. Since location transmissions must beminimized to conserve device energy, transmissions should only occurgiven geographical movement. Thus, crossing a geofence should generatesuch a transmission. Unfortunately, as crossing a geofence does notguarantee a location transmission, increasing the likelihood of atransmission is necessary.

In contrast to the known uses that surround a location with a singlegeofence, to increase the likelihood of a location transmission duringmovement, embodiments of this invention include surrounding a locationgeofence with a plurality of geofences. In one embodiment of thisinvention, a method and system of tracking a user includes determining alocation of the mobile user, automatically establishing a first geofencearound the location, and automatically establishing a plurality ofadditional geofences around the first geofence, with each geofenceincluding a boundary. A location transmission is obtained by the mobiledevice upon crossing a boundary of the first geofence or any of theplurality of additional geofences. Multiple neighboring geofences areadvantageous since they increase the likelihood of a locationtransmission as their boundaries are likewise likely to be crossed givenmovement.

FIG. 11 representatively illustrates a geofence 60 surrounding a currentlocation 62. The geofence 60 is surrounded by additional geofences 64,all within a given cellular tower transmissions cell 65. Note that partof a neighboring geofence 64′ is not fully within the cell 65, andhence, limits its benefits since a cell tower handoff by movement intocell 65′ will generate a location transmission.

Geofences are implemented as software processes. Operating systems formobile devices, such as but not limited to iOS and Android, limit thenumber of processes available to an application, and thus, the number ofgeofences is bounded. However, this limit typically exceeds the numberof geofences generated using the approach described above. Therefore,additional processes are available, and hence, additional geofences arepossible.

To increase the likelihood of a location transmission given movement, inembodiments of the invention, the remaining available processesimplement static geofences. A static geofence is not dynamicallygenerated given a new location as previously described. Rather, a staticgeofence is one that is fixed and represents those locations that arelikely to be crossed by a given user. That is, users are habitual andfrequent a limited set of locations often, for example but not limitedto, their home, office, or favorite wine or sushi bar. By learning thefrequent locations of users both individually and system wide andsetting static geofences at these locations, biasing by the individualuser, the probability of a location transmission is increased sinceadditional geofences are likely crossed.

More so, these repeated locations vary by city, county, state, country,etc., as well as by other factors such as but not limited to day andtime. Geographical and temporal presence can thus be used to vary theset of static geofences for a given user. For example, the set of staticgeofences for a given user will vary if the user is in Washington, D.C.rather than in San Francisco, Calif. Similarly, the set of staticgeofences for a given user will vary depending on the day and time. Forexample, a user frequents work on weekday mornings but frequents theirfavorite bagel shop on Sunday mornings and their favorite sushi bar onThursday evenings.

Location transmissions suffer from a margin of error. Thus, it isdifficult to precisely pinpoint and tag a location with a singletransmission. Embodiments of this invention include automatic refiningof a location of a user destination as a function of user routines, suchas established by several user visits to the destination. As timeprogresses however, and a user frequents the same location multipletimes, multiple location transmissions for the same location arerecorded. In one embodiment of this invention, as representatively shownin FIG. 12, by overlapping the transmitted location along with itsmargin of error, a more accurate location can be derived. Theoverlapping of location transmissions for a given location 70 betweenstreets 72 and within geofence 74, along with their margin of errors,represented as circles 76, yields an accurate location placement.

As shown in FIG. 12, location accuracy improves as related data arecollected. Related data, however, can, at times, be somewhat erroneous(in terms of accuracy). A non-limiting example is an entrance to ashopping mall. Such an entrance is not necessarily at the center of thecomplex. Regardless of the entrance displacement from the center of thecomplex, the entrance location can still be used to increase locationaccuracy of the mall complex since the readings for the entrance areconsistent. That is, for a given user, given mobile device, givencarrier, etc., such location recordings remain consistent, all be it,slightly erroneous. Thus, even dirty, namely potentially inaccurate,data can result in correct location identification.

Additionally, having established a location, corresponding lat-long paircoordinates can be reversed engineered, namely mapped back onto, a placename. These derived lat-long pair coordinates become yet an additionalinformation component that is used by a learning system to better refinea mapping to a named place. Machine learning, data mining, andstatistical approaches that are supervised, semi-supervised, orunsupervised can be used, as known in the art, to cross-correlate allavailable location related data.

Once determined, the user information including the location and/or theuser activities are automatically stored in a database. Embodiments ofthe MPSM method and system of this invention include a computer serverfor providing and implementing the tracking and/or social networkingservice of this invention. The computer server includes a locationmodule to determine the user location and/or a tagging module configuredto correlate manually entered user information to a user destination anda database module configured to store user information including userlocations and user activities at the user locations. For social mediaand photo sharing, the server further desirably includes a communicationmodule configured to automatically share a user activity or photo in thesocial networking service upon further user arrivals at a correspondingone of the user or community locations. The server can also include anassociation module configured to associate the user activity with thecorresponding user location and any photo taken.

Since location transmissions are needed during movement, the obviousquestion arises: when should the transmissions cease? That is, thesystem must determine when the user has arrived at a location to knowwhen to perform the automatic steps discussed above. As discussed above,GPS systems are an energy drain on a mobile device, particularly as theGPS remains on and linked with the satellites to maintain locationdetection. Keeping a GPS application operating is a drain on both theprocessor and the battery of the mobile device. This invention providesa method and executable application that conserves energy by notcontinually running during use of the mobile device.

Embodiments of the MPSM method of this invention provide an automatedmethod of tracking a mobile user that includes providing a locationmodule configured to receive location transmissions, placing thelocation module into a sleep mode, awakening the location module uponreceipt of a location transmission, and determining a location with thelocation module. These placing, awakening, and determining steps arerepeated, thereby placing the application into a sleep mode when notneeded, thereby reducing the drain on the mobile device. The applicationgoes into sleep mode when necessary or when desired, such as when theapplication is not needed, e.g., during extended movement or upon anarrival at a location. In embodiments of the MPSM method and system ofthis invention, the application can go into sleep mode whenever a timesince the device awakening exceeds a predetermined time allocation, orupon a determined rate of travel exceeding a predetermined threshold,thereby indicating extended travel.

FIG. 13 illustrates one exemplary, and non-limiting, method according toan embodiment of this invention to automatically detect arrival at adestination. The method is useful for tracking a user's location for anyof various reasons, including, for example, for safety, to provideautomated reminders, and/or to provide automated suggestions to the userbased upon the destination and/or surrounding area. The method of FIG.13 is particularly useful for implementing the method and systemdiscussed above, and can be used to implement other applications andmethod to provide energy savings compared to GPS location methods inmobile devices.

FIG. 13 includes a flow chart 200 that includes and/or represents threedistinct situations, namely, an actual arrival, rapid movement, andsporadic movement without an actual arrival. Initially, the applicationis in sleep mode. Sleep mode is a state when no processing, and hence noenergy consumption, takes place. Processing occurs once the applicationis awoken. A location transmission, such as a cell tower transmission oranother application obtaining location information, awakens theapplication in step 202. Since the application awakening occurs due to alocation transmission, the current location is known.

Once awakened, the application typically has a maximum amount of time tocomplete its processing. This limit, called time allotment, is set bythe device operating system. All processing must complete prior toexceeding the time allotment. Ideally, the application should relinquishthe processing flag back to the device operating system before theoperating system forcefully removes the application from its activequeue. Voluntarily terminating an application, namely returning it tothe sleep mode, rather than having it forcefully terminated by the hostoperating system, is consider good citizenship. In step 204, theapplication initializes two timers, namely, a timer count representingthe duration of time the process has executed since last awakening, anda stationary count representing the duration of time since the lastdetected device movement.

As time progresses and the process executes, the timer count isincremented in step 206. In one embodiment of this invention, wheneverthe application processing time exceeds the operating system timeallocation (208—YES branch), the application is voluntarily placed insleep mode 205. Note that the time allocation threshold is notnecessary, but set to support good citizenship.

Assuming that the time limit has not been reached (208—NO branch), theapplication waits for t time units in step 210. After waiting t timeunits, new current location data are obtained is step 212 and storedlocally on the device in step 214. In step 216, the current location iscompared to the previously known location. If the two locations differ(216—NO branch), the rate of travel is computed in 218. If the rate oftravel exceeds a threshold (220—YES branch), the process is desirablyand voluntarily placed in sleep mode 222. Rapid travel is unlikely toresult in an immediate or near term arrival; thus, checking locationswhile moving rapidly unnecessarily uses device energy. Eventually, theapplication process is awoken with the device moving at a slower rate.At that time, location checking is needed as an arrival might soonoccur. If or when the rate of travel is slow (220—NO branch), movementis noted in step 224, and the loop is repeated commencing with theindication that additional processing time has elapsed in step 206.

Thus far, the arrival detection process has been voluntarily placed insleep mode either due to having exceeded the self-imposed processingallotment quota which is desirably set just slightly below the operatingsystem's time limit that leads to the removal of the application fromthe active queue (208—YES branch) or having travelled too rapidly(220—YES branch). Slow travel has resulted in simply recording thelocations traveled, noting the movement exists in step 224, and awaitingeither arrival or process termination.

Arrival is determined when the same location is detected for asufficient duration of time. That is, an arrival is potentiallydetermined when the location remains the same (216—YES branch). Thestationary detection count is then incremented in step 226. If thestationary threshold is not yet exceeded (228—NO branch), theapplication waits for t time units in step 210, and the current locationis obtained in step 212 and stored locally in step 214. A sufficient andpredetermined duration at the same location eventually surpass thearrival detection threshold (228—YES Branch).

Once arrival is determined, arrival is declared in step 230, all dataregarding the prior locations visited and stored locally are compressedand sent to the back end system supporting the application in step 232.A new location checkpoint is established in step 234, and the process isplaced in sleep mode 236. From the sleep modes, the process of FIG. 13repeats upon a known location.

Compression of location data is typically performed prior to localdevice to back-end system transmission as often the location data manynot be needed at the back end. Location data may not be needed in cases,for example but not limited, during rapid travel. Although exemplifiedas having data compression occur prior to the sending of the data to theback-end, it is within the scope of this invention to compress locationdata prior to storing them locally.

All parameters described above for FIG. 13, for example t (for the timeunits), timer count, etc., are system and device dependent.Experimentation with and fine tuning of these and other parameters iswithin the scope of this invention. Also within the scope of thisinvention is the tuning of these and other parameters via the use ofmachine learning, data mining, and statistical approaches; supervised,semi-supervised, and unsupervised approaches can be used.

As discussed above, once the user has arrived at a destination, thelocation identification, user activities at the location, and/or anyproximate third party members of the user's community are determined, ifnot already known. In this way, the devices automatically continuallydetermine locations which can be used to identify any establishmentsand/or any community members at or within proximity to the location.

User activities are actions or events. Example user activities includebut are not limited to “drinking wine,” “flying,” “reading,” “attendingconference,” or “commuting.” Users specify a particular user activityeither by selecting from a provided list or by entering a different useractivity. As discussed above, the provided list is generated by storingall previously entered user activities from all systems users butbiasing the ranking of the provided activities based on context, thelocal user, their community, or a combination thereof.

All location and user activity pairs are stored in a databasecorrelating the location with the activity. Any of the many databasemanagement systems or storage schemes known in the art can serve as theplatform for this location-activity database. Furthermore, it is wellunderstood in the art that the location-activity database can store manyadditional features. For example, the user identity and date and time ofthe pair are recorded.

Over time, the database grows and contains a sufficient number of pairsto support mining. The volume of data needed to mine correlations isdependent on the mining algorithm deployed and the level of accuracyneeded. As known in the art, there are many machine learning, datamining, and statistical approaches to support mining. By using any ofthe many available such approaches, either individually or incombination, a local user activity preference per location is learned.Example learning approaches include supervised, semi-supervised, andunsupervised approaches including but not limited to Naïve Bayes, NeuralNetworks, Support Vector Machine, and Associating Mining basedtechniques. The use of proprietary mining techniques is likewise withinthe scope of this invention. Once local user preference is learned, thispreference is used to bias the aforementioned provided user activitylist.

There are many approaches to identify locations. Automated locationidentification is accomplished by periodic checking of the currentlocation. Periodicity of checking depends on, for example, the methodused to determine the location, the desired frequency of reporting,recording, and notification, and the resources available to support thechecking. Other periodicity determination approaches known in the artcan likewise be used. One approach to automate location identificationis the periodic determination of lat-long pairs via the use of a GPSdevice. An online service or a locally resident database can be used tocorrelate the GPS readings with locations. A preferred embodiment ofthis invention uses the aforementioned location database. Whenever atransmission to a connected cell tower is made, the cell coordinates ofthe transmitting device are used as a search query against the locationdatabase. If a match is detected, that location is identified. Anotherpreferred embodiment detects locations upon the crossing of geofenceboundaries as previously discussed. Note that both dynamicallydetermined geofence boundaries and static geofence boundaries detect alocation. Yet another preferred embodiment detects locations bycapitalizing on location transmissions generated by any otherapplication operating on the mobile device requesting locationinformation.

In embodiments of the MPSM method and system of this invention, localusers, unless disabled by a local user, can be provided with automatednotifications for themselves and for their community members. Thesenotifications describe locations, activities, or correlated locationsand activities for themselves and their community members. For example,unless disabled by the user, any time a user arrives at a new location,the local user and their communities can be notified of the user's newlocation. Automated location detection and notification, unlessdisabled, occurs without requiring a local user prompt.

Similarly, activity notification can be automated. Once a user arrivesat a location, a set of activities previously occurring at that locationis shared with the community or provided to the local user forinformation or sharing. If the user chooses to confirm at least one ofthese past activities, both the local user and their respectivecommunity members are notified of this at least one activity, and anyphoto taken is automatically tagged with the context information.

In another embodiment of this invention, automated notification involvesshared experiences. A shared experience is one that associates multipleusers. These associations can be passive or active. A passiveassociation is strictly informative in nature while an activeassociation requests an action. Non-limiting examples of passive sharedexperiences based on locations include: “User A is at User A′s office,as is User B” and “User A is at home as is User C.” Note that the firstexample involves multiple users at the same physical location, namelyUser A′s office, while the second example involves multiple users at thesame relative locations, namely their homes, but at different physicallocations.

Similarly, passive shared experience notifications can be based on useractivity. Non-limiting examples of passive shared experiences based onactivity include: “User A is eating lunch as is User B” and “User A isparticipating in her favorite sport as is User B.” Note that the firstexample involves multiple users participating in the same activity,namely eating lunch, while the second example involves multiple usersinvolved in similar nature of activities, namely participating in theirown favorite sport, which can be different actual activities, namelyracquetball and swimming. In both passive shared experiences based onlocation and on activity, known in the art machine learning, datamining, and statistical approaches that are supervised, semi-supervised,or unsupervised approaches can be used to correlate relative locationsand activities to physical locations and activities.

Other shared experiences can prompt for action, and are thus consideredactive. A non-limiting example of an active shared experience promptingfor action includes: “User A posted a picture when at Penn Station; youare now at Penn Station; please post a better picture?” Thus, activeshared experiences request the user to actively react. As above, activeshared experiences can be location or activity based and can be absoluteor relative. Note that it is likewise within the scope of this inventionthat individual user notifications be active and passive, in a similarmanner as described above. However, the correlation of locations andactivities both for passive and active are based strictly on thecurrent, past, or projected expected activities of the individual userrather than those of multiple users.

Typically, only changed locations and activities are notified. That is,a location or activity is not typically repeatedly identified. However,a local user can request repetitive notifications based on anytriggering condition known in the art.

Local users do not always remember to indicate a new location name orconfirm which of the possible suggested name or names the systemindicated for the given the location. As such, it is at timesadvantageous to prompt the local user for information. However, overlyaggressive prompting might annoy the user. In embodiments of thisinvention, the application non-invasively prompts the user upondetecting an unknown location for the given local user. To avoidannoyance, prompting is repeated only rarely, say twice; the number ofrepeated prompts can be set as a parameter. Similarly, to provide asense of comfort, if the back-end system recognizes the location basedon the local user's community members' naming schemes, it prompts thelocal user with guiding messages, for example but not limited to “Manyof your community members call this location The Tasting Room”.

Identification of activities associated with a given location or a givencommunity member can be additionally or alternatively automaticallyinferred in multiple ways. In embodiments of this invention, thecomputer system can automatically determine a positional destination ofa user, such as by using a mobile device discussed herein, andautomatically deduce as user information a location type and/or useractivity of the positional destination. The user information can bededuced, at least in part, based upon the destination context. Exemplarycontext information includes, without limitation, time-dependentinformation (e.g., what time of day is it?), community information(e.g., who is also there?), and/or third-party information about thepositional destination. This method, tied with automatic sharing of theuser information in a social networking service, can provide a partiallyor fully automated process for determining user location and activity,and tagging photos taken with the context information.

In one embodiment of the MPSM method and system of this invention, theautomatic deducing of the user information is based upon known orlearned user routine. As discussed above, local users typically followstandard routines. Some routines are daily, weekly, monthly, etc. Otherroutines are context dependent. Regardless of the nature of the routine,learning via any of the many statistical, machine learning, data mining,or business analytical techniques known in the art, enables predictivebehavior and automated activity and location suggestion. For example,but not limited to, if a local user always goes out to lunch at noon onevery weekday, then if an unknown location is detected on a Tuesday atnoon, then the application can suggest that this unknown location islikely a restaurant and the activity is likely eating lunch. Similarly,routine identification enables the prevention of transmissions bothpositional and informational. For example, but not limited to, if alocal user always goes to sleep at midnight on Sunday through Thursdayand awakens at 7:00 am the following day, then energy can be saved ifthe application voluntarily places itself in sleep mode during the hoursthat the local user is known to be sleeping. Additionally, routines caninvolve a sequence of activities and locations. A non-limiting exampleof a sequence of activities includes: On weekdays, Eric arrives at hisoffice at 8:00 am, drinks coffee at 10:00, develops software from 11:00am until 5:00 pm, commutes home at 5:30, and finally, arrives at home at6:00 pm.

Another location and/or activity deduction approach is by association.The automated deducing can include automatically associating a user witha second user at a positional destination. If the second user's locationand/or second user's activity is known, then the system canautomatically infer the location type and/or user activity of the firstuser from the second user location and/or activity. Consider a previousknown event such as: “Community member Sally swimming at the Somersetpool”, assuming that the Somerset pool location was previouslyidentified. As an example of automatically determining a currentactivity of community user Sam, the system identifies through locationdetermination that Sam is currently at the same location as Sally, andalso that Sally is currently at the Somerset pool. From thisinformation, possible automatically postulated associations andactivities are: “Sam is at the Somerset pool”, “Sally is swimming”, and“Sam is swimming”. Thus, it is possible to infer an activity for acommunity member from association with another community member. It iswithin the scope of this invention to use any logical inference methodsknown in the art to generate plausible associations. It is also withinthe scope of this invention to obtain confirmation of the plausiblepostulated activity by the community member, in this case Sam, by askingeither Sam or Sally or by any other means known in the art.

Desirably the computer system operating the MPSM automatically storespast user information, including past location type and/or user activityof the positional destinations of all users. User information for futurevisits to repeat positional destinations can be automatically deduced asa function of the stored past location type and/or user activity of thepositional destination. In embodiments of this invention, the system canrely on recorded previous activities of a user, a community member, orany system user at a given location to postulate on a user's activity ata given location. Past context information for past visits to thepositional destination by the user and/or community members of the usercan be compared to a current context of the user's visit to thepositional destination to deduce the user information. In oneembodiment, the system can reduce possible location types and/or useractivities as a function of the past location type and/or user activityof the positional destination.

As an example, at a given Location A, users previously studied, talked,ate, and drank. Thus, if a user's positional destination is detected asat Location A, then plausible activities postulated can be studying,talking, eating, and drinking. More so, if the given user's communitymembers only previously talked, ate, and drank, it is with a higherprobability to postulate that the given user is talking, eating, anddrinking rather than studying. Furthermore, if the given user visitedLocation A previously, but only talked and drank, then an even higherprobability is that the user is currently talking and drinking ratherthan eating and studying. It is within the scope of this invention topostulate some or all of the previously detected activities of a givenlocation. More so, it is within the scope of this invention to rankorder the activity suggestions according to the relevance of thepreviously visiting users to the given current user. As previouslydescribed, the system can request confirmation of suggested activitiesthrough the user's mobile device.

The system can additionally or alternatively reduce possible locationtypes and/or user activities as a function of the past location typeand/or user activity of the positional destination as a function of thetime of day. The system can rank possible location types and/or useractivities of the positional destination based upon known past timeperiods corresponding to the time of day of the current user visit. Forexample, again given Location A, if previous visiting users wererecorded to study one or more times during the intervals: 3:00-4:30 PMand 7:30-9:00 PM, and to drink one or more times during the intervals:4:00-7:00 PM and 8:30 PM-2:00 AM, then a current visiting user atLocation A at 3:15 PM is likely studying, at 4:15 PM is likely to beeither studying or drinking, and at 1:00 AM is likely to be drinking.More so, if the given user's community members only studied between3:15-4:30 PM then it is with a higher probability to postulate that thegiven user is studying rather than drinking at 4:15 PM. Furthermore, ifthe given user visited Location A previously but only studied, then aneven higher probability is that the user when at Location A is studying.It is within the scope of this invention to postulate some or all of thepreviously detected activities of a given location. More so, it iswithin the scope of this invention to rank order the activitysuggestions according to the relevance of the previously visiting usersto the given current user. As previously described, the system canrequest confirmation of suggested activities through the user's mobiledevice.

In embodiments of this invention, time context alone can be used topostulate activities. For example, if most days, a user is recorded tobe drinking coffee between 9:00-10:00 AM, then, without contradictoryinformation, a plausible activity postulate is that at 9:35, the user'sactivity is drinking coffee. Again, as previously disclosed, it iswithin the scope of this invention to rank order the postulated activitysuggestions according to the relevance of the previous users to thegiven current user and/or to obtain confirmation of suggestedactivities.

Additionally, it is also within the scope of this invention to rankorder the time postulates based on frequency of occurrence within thetime interval. This rank ordering applies to both location based andlocation independent time based postulates. For example, if in theinterval 4:00-4:30 PM, community members studied 25 times but drank 5times then, at 4:15, it is with a higher probability to postulate thatthe given user is studying rather than drinking.

In embodiments of the MPSM method and system of this invention, thesystem can search and/or use, if available, external, third partyinformation about the positional destination for postulating activitiesfor a given location. For example, third party vendors might provide,either free of charge or for a fee, activity information for a givensite. Consider a marketing website of a centralized homepage for agrocery store chain. Such websites are likely to contain addresses ofmany or all of the associated stores. Since these stores all supportshopping, an activity associated with these locations is shopping.Similar information can be derived or purchased from other sources suchas but not limited to commercial information repositories. Additionally,maps can be parsed. Given a location of a road, an activity of thatlocation is likely to be driving. Various and alternative third partyinformation gathering approaches and their incorporation into activityclassification and postulation can be incorporated into the method andsystem of this invention.

Suggested activity information, particularly but not limited toinformation obtained or derived from third party vendors, might beadditive or might be contradictory. Thus, combining or reconcilingpotential activities is needed. The use of voting schemes, biased basedon credibility of the source or on frequency, such as majority, or otherknown techniques, can be incorporated in the method and system of thisinvention. Note that differing suggested plausible activities mayadditive or may be contradictory. The use of techniques such as, but notlimited to, conflict resolution methods, ontology determination, andclustering, etc., can be incorporated to recognize potential conflictsand to expand classification is within the scope of this invention.

Additionally, the classification of plausible activities based onactivities occurring in the surrounding vicinity is likewise within thescope of this invention. For example, consider an unknown locationadjacent to two known locations, such as, but not limited to, twoneighboring stores or two neighboring beaches. For the neighboringstores, known activities might include shopping and strolling, while forthe neighboring beaches, known activities might include sunbathing andswimming. Given location proximity, it is within the scope of thisinvention to suggest a user's activity at the unknown location to beeither shopping and strolling or sunbathing and swimming, respectively.Confirmation can always be obtained for suggested activities and to biassuggested activities based on user familiarity and frequency ofoccurrence.

In embodiments of the MPSM method and system of this invention, localusers can opt to delay their notifications. That is, once a location isvisited or an activity occurs, a local user can opt to have thenotification of the location or activity be delayed by some period oftime. Delaying a notification provides the local user with the abilityto notify their community of the location visit or activity occurrence,but still provides the local user time to progress to the next locationor activity. As discussed above, users can also choose to automaticallyshare or not share photos taken with the digital picture frames of thisinvention.

Notifications can be complemented with correlations with other communitymembers. That is, both the local user and their respective community canbe automatically notified with a comparison. A comparison, for examplebut not limited to, can identify other community members havingpreviously conducted a specific activity or having visited a givenlocation previously. Comparisons are made by checking other communitymember locations and activities against those of the local user.Checking is performed via a search of the location-activity database. Ifa match exists within a specified or predetermined period of time, acomparison notification is made automatically. The period of time can bearbitrarily set or can follow some logical time quantum such as hour,day, week, etc.

Locations and activities are known by name. However, in addition to aname, locations and activities can have associated personal labels.Labeling locations and activities can detail familiarity to the locationand activity. User labels for locations can be surrogate names, forexample, “favorite city” for Chicago, can be songs or sound waves, forexample song words “my kind of town, Chicago is” for Chicago, can be apicture, for example “the Water Tower” for Chicago, can be a video, forexample “a panoramic view of the Chicago skyline” for Chicago, or anycombination of these and other multimedia tags supported by the localdevice. Similarly, user labels can exist for activities. For example,“favorite vice” for drinking wine, or it can be a song or sound wave,for example the song words “a bottle of red” for drinking wine, or itcan be a picture, for example, a wine bottle picture for drinking wine,or it can be a video, for example “a panoramic view of a vineyard” fordrinking wine, or any combination of these and other multimedia tagginglabels supported by the local device.

In embodiments of the MPSM method and system of this invention, localusers and community members can comment on their own and each other'slocations and activities. Comments can take any of the many multimediaforms provided by the local device. These include, but are not limitedto, text, sound, pictures, videos, or any combination thereof Multiplecomments can be made by the local user, their community, or combinationthereof In addition to stating their opinions (commenting), communitymembers can prompt for clarification. That is, by issuing “what”comments, community members request additional information on the postedlocations and activities. Additionally, user can “like” their own andeach other's locations and activities. By “liking” a location oractivity, community members express their satisfaction of theirrespective community members' presence in terms of location andactivity. Multiple community members as well as the local user can“like” a location and activity.

The MPSM method and systems of this invention can track vast data onboth the local user and their respective community members. These datacover, including but not limited to, locations, activities, and alsoindividuals both who are system users and those who are not. These datacan be stored and summarized. A summary of the local user and communitymember locations, activities, time durations involved in each of theselocations and activities, individuals who they encountered, etc., can becomputed and presented to the user. This summarization can range fromsimple statistical aggregation to advanced correlations as derived byany of the many, both individually and combined, machine learning, datamining, business analytics, and statistical techniques known in the art.

Information that can be aggregated or derived can answer, exemplary butnot limiting, questions such as: how much time a local user spent doingthings, such as, working at home, working out, walking the dog,commuting to work?; how much time a particular community member spentdoing things, such as, working at home, working out, walking the dog,commuting to work? (Note that the information derived for the communitymember is based strictly on the information that that particularcommunity member chose to share.); who are the five most commonindividuals that a particular user interacts with?; what is thelikelihood that after seeing a particular user, the given local userwould see a particular different individual?; which activities andlocations are most closely associated with each other and when are theymost likely to occur?; which three users among a given community aremost likely to visit a particular location together?

Local users can be provided with summaries of their locations, durationsat these locations, and activities at these locations. Furthermore, atthe discretion of the local user, these summaries are made available totheir community members.

The system can also generate and maintain both aggregation and derivedinformation. This information can be used to optimize suggestions toavoid obstacles, for example, but not limited to preferred routing ofcommuting path, promoted target advertising, for example but not limitedto location of nearby ice cream store for those users who frequentlyrecord “eating ice cream” as an activity, and a host of otherinformational services known in the art.

The following examples illustrate, without limitation, the abovediscussed data capturing, storing, analyzing, mining, and presentingMPSM functionalities of this invention. It is to be understood that allchanges that come within the spirit of the invention are desired to beprotected and thus the invention is not to be construed as limited bythese examples.

FIG. 14 illustrates a location summary of an individual user. As shown,two boxes are presented. The top box is a summary of where and how auser spent their last two weekend days, while the bottom box provides asummary of where and how a user spent their last five weekday days. Asshown, in both cases, the duration of time spent in a given location islisted. For example, looking at the top window, the user spent about 9hours in Georgetown in Washington and about 1 hour in O'Hare in Chicagoduring the last two weekend days. The user was obviously on travel asthe user spent less than a minute at home during the weekend.

FIG. 15 illustrates a user's transit summary, which is a summary of theuser's transit characterized by speed, namely slow, medium, and fasttravel, and when, where, and for how long did this travel occur, namely,duration and initial and terminating locations. The average speed islikewise noted. For example, the user traveled fast from Denver airportto home, a distance of 964.57 miles at an average speed of 291.29 MPH,and it took roughly 3 hours.

FIG. 16 illustrates a user's own activity summary. That is, a summary ofthe user's weekly activity is provided that includes the frequency ofand percentage and absolute time involved in the activity over the pastweek. For example, the user was at the Four Seasons twice within theweek for 3.4% of their reported time or about 4 hours.

FIG. 17 illustrates a user's time summary in comparison to theirfriends. The summary of the user's time breakdown is made in comparisonto others within their community. For example, the percentage of timethe user spent at home as compared to that of their friends is roughly14% (0.14×) versus in transit which is 1.53 times as much.

FIG. 18 illustrates a summary of a user's activities shared with others.The summary of the user's activities is shown providing an indication ofthe amount of time and activities jointly experienced. For example, theuser jointly had sushi with and visited Ophir at his home. Alsoindicated in the bottom portion of the frame are the names of one'scommunity members that the user did not see (top listing) or interactedwith (bottom listing) in the past week.

FIGS. 19 and 20 illustrate views presented to the user of a quantifiedtogether board and of a quantified individual board, respectively. FIGS.19 and 20 represent information in a more pleasing form for presentingto the user. FIG. 19 summarizes time spent and activities experiencedtogether. For example, two users spent 73 hours together this weekeating, working, and running. The last time they had drinks together wastwo weeks ago (bottom right boxes). The last time they were together was3 days ago (top left box). FIG. 20 summarizes a user's week in review.The week in review highlights the main activities and locations (tophalf). Likewise summarized are other behavior patterns includinganomalies (bottom half).

Embodiments of this invention incorporate picture characters orpictograms into the mobile positional social media domain. The methodand system of this invention allow users to define, develop,incorporate, modify, classify, and/or transmit pictograms, such asrepresenting user locations and/or activities, to community members andto global users. In one embodiment, a user of an MPSM device can selector create new one or more pictograms specific to themselves, to theircommunity, or globally, that is, to any user that has access to that oneor many particular pictograms. Users can define or redefine existingmeanings of each pictogram; a user can incorporate existing, namelyalready created and defined pictogram, into their messages; a user canassociate a picture to a given pictogram; a user can modify existingpictograms both for local and for her/his community; and/or a user canalso classify a pictogram as to its type, for example, but not limitedto, mood, activity, location, etc.

Any suitable pictogram can be added and/or used in the method and systemof this invention. One exemplary pictogram is known as Emoji. Emoji is acommonly used term that generally means picture characters or pictogram.Some Emoji representations are mapped onto Unicode representation andare thus available for use in a variety of desktop and mobile deviceapplications including the invention disclosed herein.

Using the location and/or activity determination methods discussedherein, the MPSM can determine, and possibly announce, activitiesoccurring, having occurred, or are scheduled or likely to occur usingpictograms. The pictograms can also be used to associate activities witha given location; the location of interest either being, will be, or waspreviously visited by a community member or is of relevance to a userrequest.

In one embodiment of this invention, a method and system of sharinglocations and/or activities of a user participating in an MPSM includesthe system receiving a user-defined pictogram for a destination andautomatically associating the pictogram with the destination and/or anyphotos taken. The method desirably also includes the automatic sharingof the pictogram with the photos and/or within the social networkingservice upon further user arrivals at the destination prior to receivingany additional user information. The pictogram desirably corresponds toa user activity at the destination, and can be manually selected by theuser from a list of predetermined pictograms or other photos ordrawings, etc.

Embodiments of this invention include a system that learns to associatethe pictogram with the destination and photos taken upon further visits.The pictogram can be automatically presented to the user through amobile device upon reaching the destination for confirmation and/orchanging to or selecting a new pictogram. These steps can occur forseveral visits to the destination, with the goal for automated learningand ultimately to provide an automated sharing of the pictogram for thedestination and/or with any photos. Where several pictograms have beenassociated with a destination, the several pictograms can be presentedover time for confirmation, preferably in a ranked list. In addition,the method and system can automatically determining one or more of theplurality of pictograms to share at a further arrival at thedestinations as a function of an automatically determined context of thefurther arrival, such as based upon a time of day or the presence offellow community members also at the destination, as discussed furtherherein.

The pictograms according to this invention allow for efficient pictogramsummaries of user and community activities and/or locations for anypredetermined time period, such as a day, week, month, and/or year. FIG.21 is a mobile device GUI that shows users' timelines of activities viapictograms. The pictogram timeline 140 includes pictograms that providevisualizations of a person's and their community's daily activities. Theuser activities are visually compared to what others are doing. The userand the community are represented by photographs and their names. Themost recent activity is represented by a pictogram 150 on the horizontaltimeline 140 that is closest to the user pictogram 142. The remainingpictograms 150′ for each user are showing in order of newest to oldest.The timeline can also display the timing 152 of the most recentpictogram. Feedback by others on the user's activity is also provided.In FIG. 21, one comment is currently available, as shown by the commentnotification 155.

In embodiments of this invention, photographs or other data can beassociated with the pictograms. In FIG. 21, a camera icon 154 can beassociated with, such as by overlapping, any pictogram to show when oneor more photographs or other data items are associated with thepictogram. Community users can touch or click the pictogram to displaythe associated photograph. The camera icon at the bottom of the screencan be used by the user to associate or change photos. The pictogramscan optionally be shown with the corresponding photos on the digitalpicture frame.

In another embodiment of the MPSM method of this invention, users can begrouped by common current or latest activity. As shown in FIG. 21, thefirst two community members display the same ‘house’ pictogram,indicating a common activity, albeit not necessarily at the samelocation (i.e., each is that their own home).

FIGS. 22-24 illustrate alternative pictogram summaries of a user'slocations and/or activities. FIG. 22 illustrates a summary of a user'sdaily activities via pictograms. The visualization summary shows theindividual's activities by pictogram for several days. FIG. 23illustrates a pictogram summary of a user's activities on amonth-by-month basis. FIG. 24 illustrates a histogram summary of auser's monthly activities aggregated by type and presented as avisualization of an individual's monthly activities.

FIGS. 25-27 illustrate an exemplary embodiment associating pictures withthe pictograms. FIG. 25 illustrates the option to take a photograph andassociate it with a selected pictogram. Additionally, it is possible toannotate the image with text using the annotating tool at the bottom ofthe GUI. To assist the user, the system shows or suggests a pictogramrepresentation. The selection of a new pictogram is possible if desiredor if the system suggestion is incorrect. FIG. 26 illustrates the optionto select a new pictogram to associate with the newly taken photo. FIG.26 occurs upon tapping the picture in FIG. 25. Two alternative sectionsexist in FIG. 26: a first section of suggested pictograms, such as basedupon learned location information, and a second section with pictogramswhich are globally available. The system postulates a pictogram toassociate with the picture using any of the many learning classificationalgorithms discussed herein and/or known in the art. FIG. 27 illustratesa system generated suggestion of the location associated with thephotograph. The user has the option to accept or modify the annotation.

This invention further provides a means to rapidly display and reviewuser photographs. In one embodiment, the device, such as through anexecuted application, includes and executes code instructions forproviding rapid feedback in an ergonomically convenient, intuitive userinterface. The code allows for displaying user photographs or other datain response to a plurality of user hand movements within a continuoususer swipe on a mobile device screen. User hand movements are eithercaptured via a touch screen or via gestures captured via the line ofview of an included camera.

By a user interface method, referred to as the Ownbey-scrub, or “scrub”for short, a user can hold down an icon to select and swipe through aclustered collection of pictures representing activities that occurwithin a given time span, location, with specific other users, or anyother clustering condition. Clustering of items can be accomplished inany of the many known in the art clustering techniques.

In embodiments of this invention, the scrub operates by placing a finger(and holding it) on an icon presented on a touch screen and scrubbing(without lifting off the screen) with that finger horizontally (left orright) or vertically (up or down). In response to the direction of thecontinuous finger contact/movement, quick feedback is provided to theuser in the form of rapidly displayed pictures. Left or right and up ordown represent forwards or backwards depending on user preference. Userpreference might differ due to any orientation (left or right handed) orany physical or logical conditions. By default, right to left and top tobottom is ascending order while left to right and bottom to top isdescending order. Composition movements consisting of both vertical andhorizontal can also be used. For example, a user might push and holdher/his thumb and scrub right to left, without lifting the hold,scrolling rapidly through her/his daily events from morning to night.Determination of direction of scrolling can be accomplished using anyknown in the art vector composition methods. Scaling the tracking of thescrub movement based on the number of pictures in the intended scrolland determining when such movement was performed can be done via anyknown in the art computations as used for such processing in the imageanalysis, graphics, and user interface domains. Additionally, it iswithin the scope of the invention to support voice activated commandsthat support similar scrub operations.

The invention further includes ranking the order the pictures arepresented within each scrub, and/or ranking the order the presentationof the pictograms associated with each user. Rank ordering can be basedon any of the many ordering criteria disclosed herein and/or known inthe art including but not limited to chronological time associated withthe set of activities, location of the activities ordered based on biasof locations, proximity to other users—either a specific user or anyusers within a user's community, proximity to a location where the lastphotograph was taken, and so forth.

Thus, the invention provides a digital picture frame including a cameraconnected to the frame, and a network connection module for use as adevice for displaying pictures from a user's electronic device and/orsocial media account or her or his community members' social mediaaccounts. The frame allows for efficient, automated access to photosrelevant to the viewer(s) of the frame. The automated frame allows forchanging photos for the viewer(s) without multiple manual steps.

The invention illustratively disclosed herein suitably may be practicedin the absence of any element, part, step, component, or ingredientwhich is not specifically disclosed herein.

While in the foregoing detailed description this invention has beendescribed in relation to certain preferred embodiments thereof, and manydetails have been set forth for purposes of illustration, it will beapparent to those skilled in the art that the invention is susceptibleto additional embodiments and that certain of the details describedherein can be varied considerably without departing from the basicprinciples of the invention.

What is claimed is:
 1. A picture frame, comprising: a display; a housingincluding the display on a first side and a back panel on an oppositesecond side from the display, the back panel including an asymmetricalnonplanar shape.
 2. The frame according to claim 1, wherein the backpanel comprises an oblique pyramidal shape.
 3. The frame according toclaim 2, wherein the oblique pyramidal shape is recessed into thehousing.
 4. The frame according to claim 3, further comprising a powerport within the oblique pyramidal shape.
 5. The frame according to claim1, wherein the back panel comprises four trapezoidal sections disposedabout a power port, each having a different size and shape.
 6. The frameaccording to claim 1, wherein the back panel comprises four irregulartrapezoidal sections.
 7. The frame according to claim 6, furthercomprising an off-center rotating stand centered between the fourirregular trapezoidal sections.
 8. The frame according to claim 1,further comprising a rotating stand disposed over a power port withinthe back panel.
 9. The frame according to claim 8, wherein the standrotates to sit the frame upon either of a pair of perpendicularlyabutting frame sides, the frame comprises a touch bar disposed on eachof a second pair of perpendicularly abutting frame sides, and the touchbar controls a digital photo change on the display.
 10. A picture frame,comprising: a display; a housing including the display on a first sideand a back panel on an opposite second side from the display, the backpanel including an off-center rotatable stand.
 11. The frame accordingto claim 10, wherein the back panel comprises an asymmetrical nonplanarshape.
 12. The frame according to claim 11, wherein the asymmetricalnonplanar shape comprises an oblique pyramidal shape including fourirregular trapezoidal sections surrounding a stand attachment, andfurther comprising a power port recessed within the stand attachment.13. The frame according to claim 10, wherein the back panel comprises anoff-center power port.
 14. The frame according to claim 13, wherein thepower port is within a stand attachment.
 15. The frame according toclaim 14, wherein the stand attachment comprises a two direction slidemechanism or a rotating joint.
 16. The frame according to claim 10,wherein the back panel comprises a power port, wherein the stand isdisposed over the power port.
 17. The frame according to claim 10,wherein the stand rotates to sit the stand upon either of a pairperpendicularly abutting frame sides.
 18. The frame according to claim17, wherein the frame comprises a touch bar disposed on each of a secondpair of perpendicularly abutting frame sides.
 19. The frame according toclaim 10, further comprising a stop mechanism configured to limit arotation of the stand to 90 degrees.
 20. The frame according to claim19, comprising a rotation limiter within a stand attachment on the backpanel.